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Abstract 


Contemporary  information  technology  (IT)  related  research  has  focused  on  use  as 
a  key  dependent  measure  for  valuing  IT.  By  imderstanding  the  determinants  of  IT  use, 
we  gain  descriptive  information  about  successful  IT,  and  prescriptive  information  for 
better  deploying  IT  resources.  Although  there  are  several  competing  theories  regarding 
IT  use,  research  findings  often  cite  their  inability  to  accoimt  for  temporal  changes  in 
usage  behaviors. 

This  thesis  introduces  quality  of  experience  as  a  potential  moderator  between  the 
determinants  of  use  and  actual  usage  behaviors.  A  pilot  survey  concerning  Internet  usage 
generated  potentially  relevant  items  which  were  later  refined  into  a  questionnaire 
assessing  each  item’s  relative  importance  to  perceptions  of  quality  of  experience. 

Initial  indications  suggest  10  of  the  items  represent  a  temporally  stable  and 
unidimensional  construct;  however,  this  thesis  further  examines  several  possible 
competing  explanations  for  the  results  in  order  to  motivate  potential  follow-on  research  in 
this  domain.  Fimdamental  issues  concerning  the  measurement  task  limit  the  degree  to 
which  scale  and  construct  validity  can  be  assessed.  Findings  are  also  interpreted  within 
the  context  of  IT  and  cognitive/behavioral  science  perspectives;  parallels  between  the 
obtained  results  and  expectations  based  on  these  perspectives  further  provide  for  face 
validity  of  the  quality  of  experience  construct. 


SUBJECTIVE  QUALITY  OF  EXPERIENCE  WITH  THE  INTERNET: 

ACCOUNTING  FOR  TEMPORAL  CHANGES  IN  USER  ACCEPTANCE 
OF  INFORMATION  TECHNOLOGY 

1.  Introduction 

Background 

In  the  fast-paced  and  ever-changing  world  of  information  technology  (IT), 
organizations  often  have  the  opportunity  to  leverage  newer,  more  powerful  IT  against  the 
current  business  environment,  streamlining  manual  business  processes  and  tasks, 
upgrading  current  IT  systems  to  take  advantage  of  additional  capabilities,  or  even 
changing  the  way  the  organization  does  business.  Experience  shows  the  appropriate 
match  between  IT  and  pertinent  organizational  variables  often  allows  the  organization  to 
enjoy  not  only  successful  implementation  of  the  IT  itself  (meeting  the  original  goals  for 
the  system),  but  also  an  increase  in  the  organization’s  productivity,  competitive  standing, 
market  share,  and  survivability  (Clemons  and  Row,  1988  &  1991;  Hitt,  1996). 

Experience  also  shows  the  risks  associated  with  implementing  new  IT  to  be  just  as 
real  as  (and  sometimes  more  salient  than)  the  potential  returns.  Wang  Laboratories 
learned  first  hand  even  a  company  in  the  business  of  IT  can  stiunble  when  implementing 
IT  systems — ^to  the  tune  of  $30  million,  3  years  of  wasted  development  time,  and  an 
unfinished  system  to  show  for  its  troubles  (Rifkin  and  Betts,  1988)!  Minimizing  such 
risks  means  the  IT  planner/manager  must  fully  understand  and  appreciate  all  the 
precipitating  factors  which  may  affect  the  success  of  a  new  IT  venture.  Without  this 
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understanding,  organizations  may  be  ill-equipped  to  intelligently  deploy  and  manage  their 
IT  resources,  especially  for  the  purposes  of  enhancing  organizational  effectiveness 
(Taylor  and  Todd,  1995b:  144). 

Although  a  host  of  organizational,  environmental,  and  technological  factors 
undoubtedly  contribute  to  the  eventual  success  or  failure  of  a  new  IT  system,  recent 
attention  in  the  field  of  IT  research  and  implementation  has  focused  on  the  role 
individuals  play  in  the  IT  implementation  process.  User  acceptance  (commonly 
operationalized  as  the  behavior  of  IT  “usage”),  in  particular,  has  been  theorized  to  be  a 
key  dependent  variable  for  determining  the  final  value  of  delivered  IT  systems  (Chau, 
1996;  Davis,  1989;  Taylor  and  Todd,  1995a  &  1995b).  Thus,  understanding  and 
quantifying  the  antecedents  to  user  acceptance  becomes  increasingly  important, 
especially  as  more  dollars  are  spent  throughout  business  and  industry  on  IT  systems 
which  promise  (or  have  the  potential  to  deliver)  performance  gains,  but  lie  idle  because 
they  are  not  accepted  by  the  end  users  (Bowen,  1986;  Young,  1984). 

Because  of  the  important  role  user  acceptance  is  thought  to  play  in  the  success  of 
IT  systems,  a  variety  of  models  and  constructs  have  been  proposed  to  help  explain  and 
predict  (to  varying  degrees  of  success)  user  acceptance.  Quantifying  and  validating  the 
performance  of  those  models  and  constructs  with  accurate  measurement  scales  has  been 
of  particular  interest,  I  believe,  because  the  practical  applications  of  understanding  and 
measuring  user  acceptance  are  equally  as  valuable  as  the  theoretical  insight  provided. 

Not  only  could  we  understand  often  wildly  variable  behavioral  responses  related  to  user 
acceptance  (Davis,  1989:3 19),  but  validated  measurement  scales  would  afford  IT 
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providers  indicators  of  how  to  better  tailor  their  products  and  offerings  to  the  needs  of  the 
consumer  (Taylor  and  Todd,  1995a:561).  Moreover,  organizations  could  assess  the 
potential  for  user  acceptance  between  comparative  IT  offerings  to  make  better-informed 
decisions  about  system  selection,  as  well  as  evaluate  their  current  IT  environment  for  any 
strengths  to  exploit  or  liabilities  to  address  (Davis,  1989:319). 

Unfortunately,  the  models  proposed  to  explain  and  predict  user  acceptance  are  not 
always  clear  and  simple;  competing  theories  which  explain  IT  usage  have  specified 
anywhere  from  4  (Davis,  1989)  to  as  many  as  12  (Taylor  and  Todd,  1995b)  intervening 
and  moderating  factors  which  correlate  with  demonstrated  user  acceptance.  For  example, 
Davis’  (1989)  Technology  Acceptance  Model  (TAM),  arguably  the  most  influential  of 
contemporary  research  paradigms  for  explaining  user  acceptance  of  IT  (Chau,  1996:185; 
Hendrickson  and  Collins,  1996:61;  Taylor  and  Todd,  1995a:561),  suggests  that  while 
both  perceived  usefulness  and  perceived  ease  of  use  influence  user  attitudes  concerning 
IT,  perceived  ease  of  use  is  also  acting  upon  the  user’s  level  of  perceived  usefulness.  The 
attitudes  which  develop,  in  turn,  affect  the  user’s  behavioral  intentions  to  use  a  particular 
IT  application.  However,  TAM  stipulates  that  perceived  usefulness  is  also  influencing 
behavioral  intentions  at  this  point  (a  more  thorough  discussion  of  TAM  will  follow  in  the 
next  chapter). 

The  conceptual  framework  established  by  TAM  has  repeatedly  enjoyed  validation 
across  various  users,  technologies  and  task  settings  in  both  the  academic  and  professional 
communities  (Davis,  1989;  Davis  et  al.,  1989;  Mathieson,  1991;  Taylor  and  Todd, 

1995b).  However,  the  relationships  between  TAM  variables  have  been  shown  to  vary  as 
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a  function  of  experience.  Specifically,  the  relative  influence  of  TAM  variables  in 
determining  patterns  of  user  behaviors  tends  to  vary  as  subjects  gain  more  experience 
with  IT  systems  (Taylor  and  Todd,  1995a).  This  temporal  dimension  in  particular  makes 
it  difficult  to  describe  IT  systems  in  terms  of  a  single,  stable  characterization  which  holds 
true  across  a  variety  of  user  populations — ^systems  must  be  assessed  based  on  the  relative 
effects  of  several  variables  subject  to  the  time  in  the  IT  systems  life-cycle  at  which  the 
measurements  themselves  were  taken. 

Problem  Statement 

How  do  we  account  for  the  time-dependent  fluctuations  in  user  acceptance?  Is  the 
simple  passage  of  time  enough  to  induce  such  changes  in  user  perceptions,  intentions,  and 
behaviors,  or  is  there  a  more  complex  interaction  of  factors  at  work?  Can  we  extend  the 
existing  models  of  IT  usage  in  such  a  way  so  as  not  to  invalidate  their  empirical  support, 
but  introduce  a  new  construct  into  the  fray — an  elegant  and  simple  solution  which 
captures  commonalties  in  temporally-related  aspects  of  the  user’s  IT  environment?  These 
are  the  questions  this  thesis  will  address. 

Research  Obi  ecti ves/Ouestions/Hvnotheses 

The  term  “quality”  has  been  bandied  about  professional  and  managerial  circles  for 
several  years  now.  The  Air  Force  devotes  entire  offices  and  organizational  divisions  to 
the  pursuit  of  quality.  But  what  exactly  does  quality  mean?  As  an  adjective,  we  apply 
the  term  to  just  about  everything — from  the  data  we  collect,  to  the  IT  upon  which  is 
stored  and  conveyed.  Quality  is  just  one  of  our  many  mental  schemas  which  help  us 
benchmark  and  compare  the  relative  “excellence”  of  one  occurrence  of  an  object  against 
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another.  And  like  the  Supreme  Coxart  justices  trying  to  define  pornography,  we  each  feel 
we  know  quality  when  we  see  it,  but  there  is  often  much  disagreement  as  to  a  precise 
definition.  When  it  comes  to  IT,  we  want  “high  quality”  systems,  built  with  “high 
quality”  parts,  complete  with  “high  quality”  support,  which  produce  “high  quality” 
results. 

But  do  we  want  user  acceptance?  While  this  may  seem  a  ridiculous  question 
(especially  in  light  of  the  preceding  discussion),  think  of  the  number  of  times  user 
acceptance  would  likely  be  specified  as  a  deliverable  in  an  IT  systems  contract.  Rarely 
would  the  Air  Force  award  a  contract  for  a  new  information  system  stipulating  it  must 
achieve  a  user  acceptance  rating  of  80  percent  or  better — although  maybe  it  should! 

More  often  than  not,  system  specifications  revolve  aroxmd  issues  associated  with  what  we 
might  consider  quality-related — ^hardware,  software,  or  support  features  of  the  system 
itself  Yet,  if  user  acceptance  is  not  factored  into  the  equation,  these  “quality”  issues  have 
the  potential  to  yield  less  than  optimal  solutions.  User  acceptance  is,  more  likely  than 
not,  simply  an  implicit  or  unspoken  goal  of  IT  implementation — ^we  wouldn’t  want  to 
invest  in  technology  we  know  no  one  would  use — ^however,  we  usually  strive  to  achieve 
that  goal  through  the  manipulation  of  quality-related,  system-specific  features. 

Because  we  want  our  IT  systems  to  do  and  be  so  many  things,  it  is  difficult  to 
know  exactly  what  someone  means  when  they  use  the  term  “quality”  to  describe  a 
particular  IT  system  or  application.  As  discussed  above,  “quality”  may  be  referring  to 
system-specific  aspects  such  as  efficient  use  of  code,  durability  of  the  hardware,  cost, 
help  facilities — all  of  which  may  be  part  of  a  common  mental  schema  we  have 


concerning  high  quality  information  systems.  At  some  point,  however,  the  user  has  to  sit 
down  with  the  system  and  actually  use  it.  What  influence  does  that  use  have  on  user 
acceptance?  Do  usage  experiences  shape  the  way  in  which  we  consistently  use  (hence, 
accept)  the  new  system? 

It  is  reasonable  to  assume  that  regardless  of  any  system-specific  features,  the 
nature  or  “quality”  of  the  usage  “experience”  will  mitigate  the  degree  to  which  users 
accept  (use)  a  particular  form  or  application  of  IT.  This  notion  is  consistent  with 
Fishbein  and  Ajzen’s  (1975)  theoretical  framework  for  predicting  behavior  in  which 
intervening  events  can  alter  behavioral  intentions  for  a  specific  course  of  action.  More 
important  to  realm  of  IT  usage,  those  behavioral  intentions  are  viewed  as  immediate 
antecedents  to  their  corresponding  overt  behaviors  (Fishbein  and  Ajzen,  1975:382). 
Assuming  this  model  of  behavior  is  a  valid  theoretical  framework  for  study,  the  value  of 
exploring  any  common  themes  or  aspects  within  the  intervening  events  between  user 
intentions  and  IT  usage  is  clear:  we  could  understand,  manipulate,  or  facilitate  those 
conditions  which  most  directly  influence  high  levels  of  IT  usage  behaviors. 

Furthermore,  it  is  possible  subjective  measurements  of  user  acceptance  may  well 
convey  the  notion  of  overall  “quality  of  experience,”  not  because  they  are  designed  to 
quantify  a  general  impression  of  user  acceptance;  but  because  overt  usage  behaviors 
reflect  the  net  effects  of  the  user’s  previous  experience  with  that  system  (Eagley  and 
Chaiken,  1993;  Fishbein  and  Ajzen,  1975).  This  argument  appears  to  have  some  face 
validity  in  light  of  another  theory  of  cognitive  and  behavioral  functioning:  Albert 
Bandura’s  self  efficacy  theory. 


6 


According  to  Bandura  (1995),  “people’s  beliefs  in  their  capabilities  to  manage 
environmental  demands  alfect  the  comses  of  action  they  choose  to  pursue,”  as  well  as 
“how  much  effort  they  put  forth  in  a  given  endeavor”  (Bandura,  1995:179).  Couching  the 
subject  of  IT  use  in  self  efficacy  theory,  it  seems  logical  that  the  quality  of  experience  a 
user  enjoys  or  endures  during  interactions  with  IT  could  influence  the  user’s  beliefs  about 
his  or  her  abilities  to  successfully  interact  with  IT  in  the  future.  Assuming  the 
relationship  between  self-efficacy  beliefs  and  its  resultant  behavior  is  valid,  it  follows  that 
any  factor  which  influences  self-efficacy  will  also  affect  observed  usage  behaviors,  both 
in  decision  to  use  the  IT,  and  in  usage  intensity.  Thus,  if  the  characteristics  of  quality  of 
experience  can  be  identified  and  adequately  captured  with  some  sort  of  measurement 
device,  several  such  measurements  may  well  convey  more  than  just  a  basic  understanding 
of  the  projected  or  actual  degree  of  user  acceptance — ^they  may  reflect  the  net  effects  of 
the  events  or  circumstances  which  transpired  during  previous  IT  interactions  to  reach  that 
level  of  acceptance. 

Consequently,  some  of  the  goals  of  the  present  investigation  is  to  explore  any 
similarities  in  the  events  or  circumstances  intervening  between  occasions  of  IT  use, 
establish  any  commonalties  between  users,  and  incorporate  them  into  the  notion  quality 
of  experience  as  they  relate  to  information  technology.  Key  to  achieving  these  goals  is 
the  discovery  and  exploration  of  the  various  facets  of  quality  of  experience,  and  the 
production  of  a  measurement  scale  which  best  taps  its  most  relevant  factors. 

Part  of  this  discovery  means  turning  to  past  findings  concerning  IT  usage  for  a 
reasonable  “place  to  start.”  However,  a  simple  meta-analysis  of  the  available  theories 


probably  would  not  capture  or  consider  everything  that  goes  on  in  the  minds  of  actual  IT 
users.  Questions  must  be  asked  which  probe  the  generic  “properties”  or  basic  “essence” 
of  the  experiences  between  user  and  IT,  and  determine  which  are  readily  accessible  and 
commonly  imderstood  if  communicated  in  terms  of  “quality.”  Hence,  this  investigation 
will  count  on  IT  users  to  answer  these  questions. 

Common  sense  dictates  that  the  quality  of  experience  construct  may  mean 
different  things  to  different  people.  However,  if  there  is  some  prototypical  mental  model 
for  the  “ideal”  or  “worst”  case  scenarios  of  IT  use,  then  there  should  be  some  measurable 
commonalties  between  users’  conceptualization  of  experiential  quality  in  general.  Given 
the  amount  of  research  already  performed  in  the  IT  usage  arena,  some  aspects  of  popular 
IT-related  constructs  (ease  of  use,  perceived  usefulness,  etc.)  are  likely  to  be  reflected  in 
user  responses  concerning  quality  of  experience.  However,  it  is  hoped  this  investigation 
to  bring  to  light  some  of  the  more  experience-based  or  temporally-dependent  criteria  by 
which  user’s  make  judgments  of  quality,  or  comparative  excellence,  in  IT. 

Research  Focus 

No  where  has  the  world  of  IT  seemingly  changed  faster  than  it  has  with  the 
Internet.  According  to  Fred  Briggs,  MCI's  chief  engineering  officer,  “It's  taken  us  100 
years  to  get  the  phone  network  to  the  point  it's  at.  The  Internet  will  get  to  that  same  level 
in  five  years”  (Ramo,  1997).  Aside  from  the  sheer  magnitude  of  information  accessible 
through  the  Internet,  information  on  the  Internet  has  the  potential  to  take  on  a  dynamic 
quality  as  it  can  be  created,  consumed,  shared,  and  changed  in  value  depending  upon  the 
current  user.  In  fact,  the  Internet’s  application  to  IT-related  tasks  is  potentially  limited 


8 


only  by  the  imagination  or  intentions  of  the  user  (Bose  and  Lightner,  1996:995-996). 
From  a  social  and  economic  standpoint,  a  solid  World  Wide  Web  (hereafter,  the  Web) 
presence  which  delivers  the  appropriate  information  to  the  customer  is  practically  a 
prerequisite  for  doing  businesses  today  (Kiely,  1996;  Ramo,  1997).  Moreover,  some 
believe  the  true  promise  for  the  Internet  has  yet  to  be  realized  as  it  is  only  just  beginning 
to  change  the  face  of  research  and  academic  collaboration  (Ives  and  Jarvenpaa,  1996). 

As  a  consequence,  many  organizations  are  scrambling  to  get  their  information  out 
on  the  Web,  and  spending  countless  dollars  and  man-hours  getting  their  own  people 
connected  to  reap  the  benefits  of  the  information  superhighway  (Egan  and  Pollack, 
1995:81).  But  is  it  worth  the  trouble?  Many  organizations  have  been  forced  to  curtail 
employee  Internet  access  to  prevent  loafing,  increase  focus  on  job-related  activities,  or 
avoid  inappropriate  usage  (Sunoo,  1996).  The  term  “Internet  addiction”  has  even  entered 
the  clinical  and  popular  vernacular  as  characterizing  both  excessive  and  unhealthy  levels 
of  Internet  use  (Dem,  1996;  Holden,  1997).  Unfortxmately,  the  Internet  itself  is  growing 
more  rapidly  than  our  ability  to  imderstand  or  control  it  (Schwartau,  1996:82).  For 
instance,  Internet  usage  policies  within  the  Air  Force  have  traditionally  been  somewhat 
vague,  providing  only  stopgaps  until  a  more  thorough  position  on  the  matter  is  taken. 

Both  the  promise  and  the  perils  of  Internet  usage  make  it  an  ideal  candidate  for 
investigating  the  concept  of  quality  of  experience  with  IT,  for  nowhere  does  there  seem  to 
be  such  a  marked  explosion  of  IT  usage.  The  Internet’s  relative  ubiquity  (at  least  here  in 
the  US)  also  offers  the  potential  for  a  widely  applicable  measurement  scale  upon 
validation.  However,  the  most  compelling  reason  to  focus  on  the  Internet  is  because  it. 


9 


by  itself,  is  simply  a  collection  of  connections;  there  is  no  system  per  se  to  which  you  can 
point  and  say,  “There,  that  is  the  Internet.”  As  such,  every  person’s  experience  with  the 
Internet  is  undoubtedly  influenced  by  a  number  of  different  factors:  browser,  method  of 
access,  reason  for  use,  just  to  name  a  few.  While  this  presents  a  challenge  to  construct 
development  from  the  standpoint  of  experimental  control,  it  also  highlights  what  is 
undoubtedly  the  greatest  potential  for  this  thesis:  a  chance  to  develop  a  measurement 
scale  which  consistently  captures  the  essence  of  quality  of  IT  experience  regardless  of  the 
specific  IT  application  in  question. 

In  addition,  the  Internet  is  a  form  of  IT  which  truly  has  no  task-specificity; 
although  individual  users  may  have  their  own  agendas  for  Internet  use,  the  Internet 
“proper”  has  no  explicit  task  for  which  it  was  specifically  designed  to  augment.  As  such, 
exploration  along  the  theoretical  lines  specified  above  affords  the  opportunity  to  include 
non-DoD  and  non-work  related  users  in  the  subject  pool.  Observed  patterns  within  user 
responses  will  further  strengthen  the  case  for  a  homogeneous  characterization  of  IT 
quality,  despite  the  heterogeneity  of  users  and  user  motivations. 

Methodology 

To  assess  and  quantify  user  perceptions  of  the  quality  of  their  Internet 
experiences,  instrument  development  activities  vvill  be  based  in  part  upon  the  critical 
incident  method,  as  proposed  by  Flanagan  (1954);  sample  items  will  be  generated  based 
upon  subjective  reports  of  the  “best”  and  “worst”  exemplars  of  experiential  quality  with 
the  Internet.  By  identifying  and  defining  the  dimensions  which  seem  most  salient  to  the 
expressed  perceptions  of  quality,  the  hope  is  to  generate  a  list  of  inclusive  items  which 
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adequately  tap  these  relevant  dimensions.  The  possibilities  of  item  and  scale  validation, 
as  well  as  the  notion  of  construct  validity,  will  then  be  explored  through  factor  analysis; 
however,  the  range  of  statistical  procedures  available  are  largely  dependent  upon  the 
quantity  and  quality  of  responses  received. 

Assumptions/Limitations 

Clearly,  two  of  the  biggest  assumptions  at  work  in  this  research  are  that  the 
criteria  and  circumstances  by  which  users  measure  and  assess  relative  quality  of 
experience  are  stable  across  successive  experiences  with  the  same  IT,  and  transfer  to 
other  occurrences  of  IT.  Should  these  assumptions  not  prove  to  be  the  case,  the  resulting 
construct  and  measurement  scale  would  only  be  applicable  to  the  Internet,  or  behavioral 
intentions  concerning  similar  forms  of  Intemet-like  interfaces,  with  other  temporal 
restrictions  on  applicability  to  a  single  IT  application.  Because  the  nature  of  this  research 
is  largely  exploratory,  there  is  no  way  of  knowing  ahead  of  time  whether  these  limitations 
pose  a  serious  problem  or  not.  Thus,  careful  post-hoc  consideration  will  have  to  be  made 
concerning  the  highly  variable  nature  of  individual  user  motivations,  absolute  effects  of 
Internet  experience  (duration  of  use  rather  than  any  subjective  evaluations),  and  the 
possible  confounds  associated  with  the  various  IT-based  subsystems  used  as  interfaces 
between  the  users  and  the  Internet. 

Implications 

“Quality”  is  already  a  rather  nebulous  and  loosely  applied  term.  Developing  and 
validating  a  construct  and  measurement  scale  for  quality  of  experience  may  give  us  the 
ability  to  better  capture  the  full  meaning  conveyed  when  we  say  something  has  “quality,” 


without  the  need  for  so  many  competing  dependent  measures  to  objectify  that  meaning. 
Moreover,  having  a  simple  and  inclusive  measure  by  which  to  assess  the  key  success 
factors  necessary  for  a  consistent  and  stable  characterization  of  experiential  quality  would 
provide  valuable  insight  to  the  IT  design  and  acquisition  process — a  process  in  which  a 
number  of  IT  systems  alternatives  are  often  assessed  and  compared  before  selection  and 
implementation.  Quality  of  experience  may  also  go  a  long  way  towards  explaining  user 
acceptance  in  some  cases  of  IT,  and  rejection  in  others,  by  giving  us  clues  as  to  what  sorts 
of  issues  are  important  to  IT  users  during  their  interface  with  the  system  itself 
Preview 

In  the  chapters  which  follow,  specific  theoretical  justification  for  investigating  the 
notion  of  quality  of  experience  vvill  be  explored.  Emphasis  vdll  be  placed  on  the  many 
competing  theories  of  the  determinants  of  IT  usage  behavior,  including  the  Theory  of 
Reasoned  Action,  TAM  and  its  subsequent  modifications,  and  the  role  experience  plays 
within  the  theoretical  relationships  presented.  The  sxuwey  procedures  and  subsequent 
analysis  used  in  the  current  investigation  will  also  be  discussed,  to  include  steps  taken  to 
generate  sample  scale  items,  scale  refinement  activities,  and  statistical  analysis  of  the 
resultant  data.  Finally,  interpretation  and  discussion  will  follow  closely  in  line  with  the 
obtained  results  which  hopefully  will  provide  clues  as  to  reconciling  any  deviations  not 
otherwise  expected  or  suggested  in  the  prevailing  research  literature. 
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II.  Literature  Review 


When  deciding  what  dependent  measure  to  use  when  evaluating,  comparing,  or 
even  discussing  IT,  the  manager/researcher  finds  himself  in  the  imenviable  position  of 
having  almost  as  many  measures  fi’om  which  to  pick  as  there  have  been  studies  devoted 
to  IT  implementation  itself  (DeLone  and  McLean,  1992:61).  However,  one  recvirring 
theme  in  the  field  of  IT  research  and  implementation  has  focused  on  a  particular  aspect  of 
IT:  use.  Use  has  been  a  key  dependent  measure  in  a  multitude  of  cases  and  studies 
concerning  IT.  The  following  analysis  will  trace  the  development  of  IT  use  as  a  central 
concern  of  IT-related  study  and  practice,  and  explore  the  theoretical  and  practical 
importance  of  understanding  those  factors  which  influence  or  determine  IT  use  itself. 
Historical  Perspectives 

A  number  of  measures  have  been  proposed  to  describe  what  it  means  to  have 
“successful”  IT  implementation.  Hitt  and  Brynjolfsson  (1996)  studied  the  “value”  of 
delivered  IT  in  terms  of  economically  quantifiable  estimates  of  business  productivity, 
profitability  and  consumer  surplus.  Although  absolute  profitability  was  not  found  to  be 
strongly  related  to  IT,  productivity  was  positively  influenced  by  IT  implementation.  In 
addition,  average  consumer  surplus  (over  total  costs)  increased  between  $2  to  $7  billion 
per  year,  suggesting  that  the  benefits  firms  enjoyed  fi-om  their  IT  investments  were 
substantial  enough  to  be  passed  along  to  the  consumer  (Hitt  and  Brynjolfsson,  1996:136). 
Unfortunately,  such  dollar  estimates  of  IT  systems  success  are  only  descriptive  in  nature; 
they  do  little  to  explain  why  a  certain  application  of  IT  might  succeed  in  one  situation  but 
fail  in  another. 
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Characterizing  IT  Systems  Success 


In  their  “Quest  for  the  Dependent  Variable,”  DeLone  and  McLean  (1992) 
reviewed  180  studies  relating  to  information  systems  success.  In  their  final  analysis, 
DeLone  and  McLean  proposed  a  taxonomy  for  categorizing  IT  systems  success  along  six 
interdependent  dimensions:  system  quality,  information  quality,  use,  user  satisfaction, 
individual  impact,  and  organizational  impact;  their  model  is  presented  in  Figure  1 . 


Although  some  economic  variables  similar  to  those  addressed  by  Hitt  and  Brynjolfsson 
(1996)  are  included  in  the  Information  Systems  Success  Model  imder  organizational 
impact  (DeLone  and  McLean,  1992:74-75,  79),  they  were  only  considered  after  assessing 
the  impact  of  system  attributes  on  the  users’  psychological  appraisal  of  the  system  itself, 
usage  behaviors,  and  an  estimate  of  the  “impact”  the  system  will  have  on  user 
performance. 

User-Centered  Thought 

An  emergent  theme  in  DeLone  and  McLean’s  (1992)  study  was  that  even  when  IT 
success  was  operationalized  in  terms  of  system  attributes  or  organizational  factors,  the 
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picture  was  incomplete  without  considering  the  interdependence  of  those  factors  with  the 
roles  played  by  system  users.  For  example,  a  particular  system  might  be  rated  high  on 
some  objective  measures  of  system  quality  or  information  quality,  but  those  factors  alone 
could  not  adequately  describe  systems  success  across  other  situations.  However,  those 
systems-centered  factors  did  appear  to  influence  users’  satisfaction  with  that  system  and 
their  subsequent  usage  behaviors.  The  culmination  of  these  effects  on  the  users  were 
found  to  be  “direct  antecedents  of  individual  impact,”  which  would  eventually  manifest 
themselves  at  an  organizational  level  (DeLone  and  McLean,  1992:85, 87).  Thus,  even 
though  the  Information  Systems  Success  Model  “reads”  from  system  to  user  to 
organization  (for  xmderstanding  IT  systems  success),  the  “hub”  of  activity  within  the 
model  is  the  user. 

This  sentiment  is  echoed  by  the  fact  that  many  studies  within  the  field  of  IT 
systems  implementation  do  focus  primarily  on  user-centered  measures,  specifically,  user 
acceptance  or  usage  behaviors  (Davis,  1989:319).  Why  would  this  be  the  case?  Why  has 
use  or  usage  behavior  become  such  a  key  dependent  variable  for  study,  especially  when 
none  of  the  six  factors  identified  in  the  Information  Systems  Success  Model  (including 
use)  were  foimd  to  be  intrinsically  “better”  measures  for  IT  systems  success  than  any 
other  (DeLone  and  McLean,  1992:80)? 

The  answer  may  simply  be  one  of  practicality.  System-specific  issues  like  quality 
of  information  or  system  quality  are  very  difficult  to  define  and  measure  consistently 
across  situations.  Attributes  of  interest  could  include  lines  of  code,  on-line  help  facilities, 
hardware  features,  information  organization  or  presentation,  media  richness,  and  a  host  of 
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other  factors  which  undoubtedly  vary  in  importance  from  one  situation  or  IT  application 
to  the  next.  Similarly,  factors  such  as  individual  or  organizational  impact  are  open  to  a 
wide  number  of  interpretations  depending  upon  the  context  of  the  measurement.  Even 
user  satisfaction  can  have  a  number  of  potential  measures  and  each  one  be  appropriate  for 
the  situation  at  hand.  With  so  many  choices  for  characterizing  IT  systems,  the  root 
causes  of  IT  systems  success  might  simply  get  lost  or  confused  amidst  consideration  of 
the  study  objective,  organizational  context,  specific  system  aspect  under  investigation, 
research  method,  and  level  of  inquiry  or  analysis  (DeLone  and  McLean,  1992:80). 

The  Pivotal  Role  of  IT  Use 

IT  use  has  recently  gained  interest  as  a  phenomenon  in  its  own  right  (Chau, 
1996:185;  Mathieson,  1991:173;  Taylor  and  Todd,  1995b:144).  This  line  of  study  has  a 
certain  intuitive  appeal;  system  features  will  make  little  difference  if  the  resulting  IT  is 
not  used,  usage  undoubtedly  reflects  some  degree  of  user  satisfaction,  and  usage  patterns 
invariably  affects  the  impact  IT  has  on  individuals  and  the  organization.  Therefore, 
exploring  IT  usage  (and  its  determinants)  allows  not  only  a  descriptive  understanding  of  a 
successful  IT  system,  but  also  prescriptive  information  for  how  to  better  deploy  IT 
resources  in  an  organization  (Taylor  and  Todd,  1995b:145).  Quite  simply,  understanding 
IT  usage  behaviors  means  understanding  why  a  potential  user  might  or  might  not  use  a 
particular  IT  system  or  application. 

In  searching  for  an  adequate  explanation  of  IT  usage,  a  preponderance  of 
contemporary  IT  usage  literature  (Adams,  Nelson,  and  Todd,  1992;  Chau,  1996;  Davis, 
Bagozzi  and  Warshaw,  1989;  Hendrickson  and  Collins,  1996;  Lederer,  Maupin,  Sena  and 
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Zhuang,  1997;  Mathieson,  1991;  Szajna,  1996;  Taylor  and  Todd,  1995a  &  1995b)  cite 
Davis’  (1 989)  TAM,  in  one  form  of  another,  as  a  viable  means  of  explaining  user 
acceptance.  Although  many  of  these  studies  will  be  addressed  in  due  course,  it  is 
important  to  first  imderstand  the  theoretical  vmderpinnings  of  the  TAM  before 
appreciating  what  the  present  investigation  brings  to  bear  on  the  subject  of  user 
acceptance  and  usage  behaviors. 

The  Theory  of  Reasoned  Action 

Typically,  when  we  say  “usage,”  what  we  are  referring  to  are  user  behaviors, 
observable  acts  in  which  the  user  interacts  with  the  IT  system.  As  stated  above,  TAM 
was  conceived  to  explain  IT  usage.  Much  of  TAM  was  developed  fi'om  the  early  works 
of  Fishbein  and  Ajzen  (1975).  Although  not  specific  to  IT,  Fishbein  and  Ajzen’s  (1975) 
Theory  of  Reasoned  Action  (TRA)  was  a  very  popular  paradigm  for  understanding 
behavior  in  general,  and  a  large  body  of  research  has  accumulated  supporting  it  (Davis  et 
al.,  1989:985).  A  graphical  representation  of  TRA  is  presented  in  Figure  2. 


Fishbein  and  Ajzen  (1975)  argued,  despite  the  reigning  theory  and  research  of  the 
day,  that  overt  behavior  was  first  and  foremost  a  consequence  of  intention  to  perform 
behavior;  previous  theory  posited  that  behavior  could  be  imderstood  fully  by  the 
influence  of  beliefs  or  attitudes  alone  (Fishbein  and  Ajzen,  1975:510).  According  to 
TRA,  behavioral  intentions  were  themselves  shaped  jointly  by  attitudes  about  the 
behavior,  and  other  normative  factors.  Attitudes  about  a  behavior  were,  in  turn, 
influenced  by  a  person’s  various  beliefs  about  the  behavior  in  question.  TRA  advanced 
the  notion  that  attitudes,  beliefs,  and  intentions  all  play  integral  parts  in  shaping 
behaviors,  but  the  distinction  between  these  factors  were  necessary  to  maintain  if  a  clear 
understanding  of  overt  behaviors  was  to  be  achieved  (Fishbein  and  Ajzen,  1975:510). 

Implicit  in  TRA  was  the  fact  that  behavior  was  the  result  of  internal  psychological 
variables  and  their  interrelations.  External  variables  also  played  a  part  in  shaping 
behavior,  but  did  so  only  indirectly  through  their  impact  on  beliefs,  attitudes  or  intentions 
(Davis  et  al.,  1989:984).  However,  many  issues  of  interest  associated  with  IT  study  or 
implementation  involve  external  variables  such  as  system  attributes  and  features,  business 
climate,  organizational  context,  and  the  nature  of  the  task,  as  well  as  internal  variables, 
such  as  user  behavior  or  cognitive  and  affective  evaluations  of  specific  IT.  Therefore, 
adequately  modeling  IT  user  behaviors  vdthin  the  context  developed  by  Fishbein  and 
Ajzen  (1975)  required  investigation  of  pertinent  aspects  of  internal  perceptual  processes, 

‘  i 

and  the  establishment  of  a  fundamental  set  of  external  variables  which  most  directly  '' 

influenced  those  perceptual  processes  (Davis  et  al.,  1989:985). 
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The  Technology  Acceptance  Model 


Davis’  (1989)  early  work  exploring  some  of  these  behavior-related  constructs 
actually  centered  aroxmd  validating  measurement  instruments  for  reliably  explaining  and 
predicting  IT  user  acceptance.  His  investigations  focused  on  what  he  considered  to  be 
two  especially  important  determinants  for  IT  user  behaviors:  perceived  usefulness  and 
perceived  ease  of  use.  Perceived  usefulness  was  defined  as  “the  degree  to  which  a  person 
believes  that  using  a  particular  system  would  enhance  his  or  her  job  performance”; 
perceived  ease  of  use  was  defined  as  “the  degree  to  which  a  person  believes  that  using  a 
particular  system  would  be  free  of  effort”  (Davis,  1989:320).  Results  obtained  from 
Davis’  (1989)  measurement  scales  (the  scales  themselves  proved  empirically  strong  for 
psychometric  evaluation)  did  indicate  that  the  theoretical  constructs  of  perceived 
usefulness  and  perceived  ease  of  use  were  significantly  related  to  self-reports  of  usage 
behavior,  although  perceived  usefulness  was  foimd  to  be  more  strongly  related  to  usage  in 
both  of  his  studies.  In  addition,  regression  analyses  indicated  perceptions  of  ease  of  use 
were  likely  antecedents  for  perceptions  of  usefulness,  as  opposed  to  a  parallel  influence 
on  system  usage  (Davis,  1989:319,  334). 

Using  the  TRA’s  conceptual  framework  for  understanding  behavior,  Davis  et  al. 
(1989)  adapted  and  refined  the  belief-attitude-intention-behavior  relationships 
specifically  for  modeling  user  acceptance  of  IT  systems.  In  Davis’  own  words,  the  goal 
of  TAM  was  to  “provide  an  explanation  of  the  determinants  of  computer  acceptance  that 
is  general;  capable  of  explaining  user  behavior  across  a  broad  range  of  end-user 
computing  technologies  and  user  populations;  while  at  the  same  time  being  both 
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parsimonious  and  theoretically  justified”  (Davis  et  al.,  1989:985).  A  graphical 
representation  of  Davis  et  al.’s  TAM  is  presented  in  Figure  3. 


According  to  TAM,  the  beliefs  of  perceived  usefulness  and  ease  of  use  are  the 
most  relevant  concerns  for  understanding  IT  user  behaviors.  Evident  in  the  diagram 
above,  the  fully  articulated  TAM  allows  for  more  complex  interactions  (than  TRA) 
between  usefulness,  ease  of  use,  usage,  and  other  relevant  factors  which  determine  user 
behavior.  Specifically,  TAM  formally  acknowledges  the  effects  of  external  variables  on 
user  beliefs  as  part  of  the  perceptual  processes  which  determine  behavior.  Unlike  TRA, 
TAM  does  not  treat  all  beliefs  “equally.”  According  to  TRA,  all  relevant  beliefs  affecting 
behavior  are  summated  into  a  single  construct;  TAM  treats  the  specific  beliefs  of 
perceived  usefulness  and  ease  of  use  as  fundamentally  different  constructs  (Davis  et  al., 
1989:988).  In  addition,  TAM  posits  the  parallel  influence  of  perceived  usefulness  on 
both  attitude  nwc?  behavioral  intention — ^that  attitudes  only  partially  mediate  the 
relationship  between  beliefs  and  intentions.  By  way  of  contrast,  TRA  assumes  these 
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influences  to  be  serial,  whereby  beliefs  directly  influence  attitude,  and  attitude,  in  turn, 
directly  influences  intention  (thus,  attitudes  fully  mediate  belief-intention  relationships). 

Davis  et  al.’s  (1989)  goal  was  to  see  how  well  TAM  explained  user  behavior 
versus  the  competing  explanation  provided  by  TRA.  Their  study  involved  the  use  of  a 
single  subject  pool  (MBA  students)  and  a  single  novel  IT  application  (a  word  processing 
package)  introduced  and  used  over  a  relatively  short  period  of  time  (14  weeks).  As 
postulated  by  both  TAM  and  TRA,  intentions  appeared  to  be  the  direct  antecedent  to 
overt  behaviors;  no  other  TAM  or  TRA  variable  significantly  effected  use  beyond  any 
mediating  effects  of  behavioral  intentions  (Davis  et  al.,  1989:992).  Nevertheless,  TAM 
explained  more  of  the  variance  in  behavioral  intentions  than  TRA  at  either  time  zero  or 
after  repeated  application  use,  while  TRA’s  subjective  norms  were  not  found  to  have 
significant  effect  at  either  time  (Davis  et  al.,  1989:993).  A  somewhat  similar  pattern  of 
results  was  also  observed  for  determinants  of  attitudes,  with  TAM  explaining  more 
attitudinal  variance  over  time  than  did  TRA  (Davis  et  al.,  1989:994).  Several  other 
interesting  findings  provided  mixed  support  for  both  TAM  and  TRA,  most  of  which  are 
beyond  the  scope  of  the  present  study. 

However,  one  set  of  findings  which  provided  a  few  kernels  of  theoretical 
justification  for  the  present  study  were  the  temporal  changes  observed  in  the  influence  of 
the  usefulness  and  ease  of  use  constructs.  Davis  et  al.  (1989)  foxmd  usefulness  not  only 
had  very  strong  effects  on  behavioral  intentions,  but  the  magnitude  of  effect  increased 
over  time.  Although  contrary  to  TAM  (and  prior  research  findings),  ease  of  use  was  also 
found  to  have  a  very  strong  direct  effect  on  intentions,  but  only  at  the  start  of  application 
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use — ^after  repeated  application  usage,  ease  of  use’s  effect  was  entirely  indirect  through 
its  influence  on  usefulness  (Davis  et  al.,  1989:994). 

Results  of  this  study  support  the  hypothesis  that  the  relative  importance  of  ease  of 
use  and  usefulness  varies  as  a  function  of  time.  Indeed,  this  would  not  be  the  last  time 
temporally  (or  experientially)  related  issues  would  be  considered  in  the  context  of  TAM 
(Chau,  1996;  Szajna,  1996;  Taylor  and  Todd,  1995a).  However,  the  fundamental 
principles  upon  which  TAM  was  based  were  generally  supported  in  that: 

(1)  People’s  computer  use  can  be  predicted  reasonably  well  from  their 
intentions. 

(2)  Perceived  usefulness  is  a  major  determinant  of  people’s  intentions  to 
use  computers. 

(3)  Perceived  ease  of  use  is  a  significant  secondary  determinant  of 
people’s  intentions  to  use  computers.  (Davis  et  al.,  1989:997) 

Validating  TAM 

These  fundamental  principles  were  later  tested  in  Hendrickson  and  Collins’ 

(1996)  study  of  spreadsheet  and  word  processing  application  usage  in  college  students. 
Hendrickson  and  Collins  (1996)  explored  possible  variations  in  the  nature  of  the  ease  of 
use-usefulness-usage  relationship — ^behavioral  intentions  were  omitted  from 
consideration  in  lieu  of  direct  reports  of  system  use  (Hendrickson  and  Collins,  1996:63). 
Three  different  relationships  between  these  factors  were  tested:  one,  ease  of  use 
indirectly  affecting  usage,  mediated  by  usefulness;  two,  parallel  effects  of  ease  of  use  and 
usefulness  on  usage;  and  three,  direct  and  indirect  influence  of  ease  of  use  on  usage,  with 
another  direct  effect  between  ease  of  use  and  usefulness.  A  graphical  representation  of 
the  competing  relationships  is  shown  in  Figure  4. 
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Figure  4.  Competing  Theories  of  Usefulness/Ease  of  Use  Effects  on  Usage 
(Hendrickson  and  Collins,  1996:63) 

Thorough  a  process  of  structural  equation  modeling,  Hendrickson  and  Collins’ 
(1996)  results  indicated  the  fully  expanded  ease  of  use-usefulness-usage  relationship 
provided  the  best  fit  to  the  data  gathered  in  their  study.  In  addition,  the  magnitude  of  the 
relationship  between  ease  of  use  and  usage  appeared  much  smaller  than  either  the 
usefulness-usage  relationship,  or  the  ease  of  use-usefulness  relationship.  Although  the 
authors  acknowledged  the  limitations  of  their  investigation  in  that  they  employed  an 
abbreviated  version  of  TAM  (attitudes  and  intentions  were  not  addressed  at  all),  their 
findings  nonetheless  provided  further  support  for  the  basic  factor  interrelations  proposed 
by  TAM,  as  well  as  justification  for  continuing  to  address  ease  of  use  and  usefulness  as 
key  perceptual  mechanisms  for  determining  IT  systems  use. 

Similar  findings  were  observed  earlier  in  Adams  et  al.’s  (1992)  replication  of 
Davis’  original  user  acceptance  and  usage  investigations.  However,  Adams  et  al.  (1992) 
sought  to  extend  the  context  of  Davis’  usefulness  and  ease  of  use  constructs  in  two 
respects.  In  their  first  field  study,  Adams  et  al.  (1992)  examined  usefulness  and  ease  of 
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use  across  very  heterogeneous  user  groups  (from  10  different  organizations  at  varying 
organizational  levels)  between  relatively  similar  technologies  (voice  mail  and  electronic 
mail).  By  way  of  contrast,  Davis’  (1989)  field  study  explored  the  usefulness  and  ease  of 
use  constructs  within  a  homogenous  user  group  (employees  at  IBM)  across  two 
heterogeneous  technologies  (electronic  mail  versus  a  file  editing  program).  Adams  et  al. 
(1992)  hoped  their  findings  would  validate  Davis’  usefulness  and  ease  of  use  scales 
across  different  situations,  test  the  discriminant  validity  of  the  usefulness-ease  of  use 
relationships  between  similar  technologies  (where  it  was  assumed  similar  ratings  were 
likely  to  be  obtained),  and  test  the  convergent  validity  of  usefulness  and  ease  of  use  as 
determinants  for  the  same  dependent  variable:  usage  (Adams  et  al.,  1992:228). 

In  their  second  study,  Adams  et  al.  (1992)  examined  differences  in  user 
perceptions  between  three  “leading”  software  packages  (WordPerfect,  Lotus  1-2-3,  and 
Harvard  Graphics)  based  on  the  assumption  that  their  respective  market  leadership  should 
translate  into  relatively  high  ratings  of  usefulness  and  ease  of  use  (Adams  et  al., 
1992:228).  While  Davis’  (1989)  lab  study  examined  the  ability  of  usefulness  and  ease  of 
use  to  discriminate  between  alternative  application  selections  for  a  similar  task  (business 
graphics),  Adams  et  al.  (1992)  investigated  the  degree  to  which  measurements  of 
usefulness  and  ease  of  use  could  adequately  discriminate  between  technology  alternatives 
all  thought  to  be  high  on  the  same  constructs.  Again,  the  strength  of  the  relationships 
between  usefulness,  ease  of  use,  and  application  usage  was  also  addressed. 

The  results  of  both  studies  were  (predominantly)  favorable.  Discriminant  validity 
of  the  usefulness  and  ease  of  use  measurement  scales  was  demonstrated  both  for 
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technologies  which  support  similar  functions  (electronic  mail  and  voice  mail  for 
organizational  communication),  and  technologies  which,  because  of  their  popularity, 
should  be  rated  similarly  high  for  both  usefulness  and  ease  of  use  (Adams  et  al., 

1992:236, 242).  More  importantly,  usefulness  and  ease  of  use  were  found  to  be 
significantly  correlated  with  reported  usage.  In  addition,  Adams  et  al.  (1992)  found 
significant  intercorrelation  between  the  usefulness  and  ease  of  use,  further  supporting  the 
merits  of  examining  both  factors  as  determinants  of  system  use  (Adams  et  al.,  1992:233). 

Unfortunately,  the  exact  nature  of  the  usefulness-ease  of  use-usage  relationship 
was  not  consistent  (with  TAM  or  each  other)  across  the  two  studies.  In  their  first 
investigation,  Adams  et  al.  (1992)  fotmd  ease  of  use  and  usefulness  strongly  related,  with 
usefulness  more  strongly  related  to  usage  than  ease  of  use  (Adams  et  al.,  1992:237). 

These  findings  were  consistent  with  those  reported  earlier  in  Davis  (1989)  and  Davis  et 
al.  (1989),  and  provided  strong  and  convincing  support  for  the  perceptual  processes 
posited  by  TAM  to  determine  user  behavior. 

In  their  second  study,  however,  three  very  different  relationships  emerged.  In  one 
case,  usefulness  was  not  found  to  be  significantly  related  to  use,  while  ease  of  use  was 
significantly  related  to  both  usefulness  and  actual  usage.  In  a  second  case,  usefulness  was 
significantly  related  to  use,  but  the  relationship  between  ease  of  use  and  usage  was 
actually  negative!  In  still  another  case,  ease  of  use  was  strongly  related  to  usage  while 
the  usefulness-usage  relationship  was  not  significant  (Adams  et  al.,  1992:239-242). 

Attempting  to  reconcile  these  inconsistent  findings,  Adams  et  al.  (1992)  suggested 
a  variety  of  factors  as  possible  confounds  including  user  experience,  perceptions  of 


captive  use  (compulsory  use  due  to  lack  of  alternatives  or  de  facto  standards),  poor 

measures  of  usage  (subjective  self  reports  versus  objective  measures  of  usage  behavior), 

and  the  nature  of  the  software  packages  used  in  the  study  (Adams  et  al,  1992:242-244). 

Though  these  explanations  were  speculative  and  perhaps  could  not  explain  all  of  the 

observed  differences  between  the  relative  importance  of  usefulness  and  ease  of  use 

(Adams  et  al.,  1992:243),  the  value  of  these  inconsistent  results  in  imderstanding  user 

behaviors  was  articulated  in  the  final  analysis: 

These  studies  show  that  the  relationship  of  these  two  constructs  to  usage  is 
perhaps  more  complex  than  is  typically  postulated.  It  may  be  that  a 
variety  of  factors. .  .may  mediate  the  relationship  between  ease  of  use  and 
usage.  As  indicated  in  Study  2,  usage  may  influence  perceptions  of  ease 
of  use.  Future  research  should  begin  to  examine  some  of  these  mediating 
effects  to  determine  the  extent  to  which  ease  of  use  and  usefulness  are 
directly  related.  Ideally  studies  will  provide  tests  of  competing  models. 

(Adams  et  al.,  1992:245) 

These  final  words  proved  prophetic  as  a  number  of  studies  have  not  only  tested  the 
efficacy  of  TAM  in  its  ability  to  model  the  relationships  between  ease  of  use,  usefulness, 
and  actual  use,  but  also  several  alternative  models  for  understanding  IT  user  behaviors 
(Chau,  1996;  Hendrickson  and  Collins,  1996;  Mathieson,  1991;  Szajna,  1996;  Taylor  and 
Todd,  1995a  &  199b). 

TAM  versus  The  Theory  of  Planned  Behavior 

Mathieson  (1991),  similar  to  the  previous  work  of  Davis  et  al.  (1989),  compared 
TAM’s  explanatory  capabilities  for  user  behaviors  to  that  of  another  more  general 
behavioral  model,  in  this  instance,  the  Theory  of  Planned  Behavior  (TPB).  On  a 
fundamental  level,  Mathieson  was  naturally  concerned  vvdth  which  model  “best” 
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explained  IT  user  behaviors.  However,  the  investigation  also  addressed  the  notions  of 
utility — ^the  degree  to  which  the  models  provide  useful  information  about  user  behaviors, 
regardless  of  predictive  power;  and  practicality — ^which  model  was  easiest  (cheapest)  to 
apply  (Mathieson,  1991:174). 

TPB  grew  out  of  the  conceptual  framework  of  the  belief-attitude-intention- 
behavior  relationships  established  in  Fishbein  and  Ajzen’s  (1975)  initial  work  with  TRA 
(the  development  of  TPB  was  discussed  in  Ajzen  (1985  &  1989),  but  is  beyond  the  scope 
of  the  present  investigation).  TPB  added  much  in  the  way  of  perceptual  mechanisms 
which  eventually  influence  behavior  over  those  identified  in  TRA.  For  example,  TPB 
introduced  new  factors  which  influenced  behavioral  intention:  perceived  behavioral 
control,  control  beliefs,  and  perceived  facilitation. 

Basically,  these  factors  relate  to  a  person’s  beliefs  that  he  or  she  has  the  skills, 
resources,  or  opportunities  necessary  to  carry  out  the  behavior,  as  well  as  an  assessment 
of  the  importance  of  those  resources  for  the  achievement  of  the  behavioral  outcome 
(Mathieson,  1 991 : 1 76).  The  relative  strength  of  these  new  factors  lay  in  the  fact  that 
control  beliefs  could  be  internal  to  the  person  (for  example,  they  lack  the  skills  or  abilities 
to  successfully  use  a  new  IT  system)  or  external  to  the  situation  (for  example,  high 
network  traffic  makes  use  of  a  new  IT  system  difficult),  thus  giving  the  TPB  more 
specificity  than  TAM  in  identifying  determinants  of  behavioral  intention  (Mathieson, 

1991 : 1 77).  TPB  is  shown  graphically  in  Figure  5. 


27 


Mathieson  (1991)  tested  both  TPB  and  TAM  to  see  which  model  better  predicted 
students’  decisions  about  using  a  spreadsheet  program  (one  of  three  alternatives)  or 
calculator  to  solve  a  difficult  mathematical  task.  In  this  instance,  obtained  results  were 
consistent  with  the  theoretical  relationships  posited  by  TAM.  Ease  of  use  explained  a 
high  degree  of  variation  in  usefulness,  and  both  ease  of  use  and  usefulness  contributed  to 
attitude.  Like  the  previous  results  reported  in  Davis  (1989)  and  Davis  et  al.  (1989), 
usefulness  was  found  to  be  a  very  strong  determinant  of  behavioral  intention — ^much 
more  so  than  attitude;  and  attitudes  were  affected  more  by  usefulness  than  ease  of  use 
(Mathieson,  1991:184). 

Overall,  TAM  was  able  to  accoxmt  for  more  attitude  variability  and  slightly  more 
variability  in  behavioral  intention  than  TPB.  TAM  was  also  an  easier  model  to  test  than 
TPB,  partially  because  Davis  (1989)  had  already  developed  and  validated  empirically 
sound  instruments  while  new  measures  of  belief  have  to  be  developed  for  each  new 
context  examined  by  TPB  (Mathieson,  1991:187).  From  a  subjective  standpoint,  TPB 
seemed  to  produce  more  specific  information  about  the  determinants  of  user  behaviors 
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than  TAM.  For  example,  TAM  offered  an  appraisal  of  the  users’  perceptions  of  system 
usefulness,  TPB  indicated  which  specific  outcomes  were  not  being  achieved  via  system 
use;  thus,  TPB  provided  “more  information  about  the  factors  users  consider  when  making 
their  choices”  (Mathieson,  1 99 1 : 1 88).  Similar  to  the  observations  of  Adams  et  al.  (1992), 
Mathieson  (1991)  found  the  TAM  to  be  relatively  resilient  to  experimental  scrutiny,  but 
perhaps  incomplete  in  some  respects  for  fully  explaining  user  behavior. 

Taylor  and  Todd  (1995b)  observed  nearly  identical  results  in  their  evaluation  of 
TAM  and  competing  theories  of  IT  systems  use.  Taylor  and  Todd’s  (1995b) 
investigation  of  computer  resource  center  users  again  pitted  TAM  against  the  explanatory 
power  of  TPB;  however,  they  also  introduced  a  decomposed  TPB,  less  parsimonious  than 
TPB,  but  accounting  for  even  more  situational  and  personal  variables  than  either  model. 
The  decomposed  TPB  is  shown  in  Figure  6. 
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Figure  6.  Decomposed  Theory  of  Planned  Behavior  (Taylor  and  Todd,  1996b:146) 


Similar  to  the  methodology  reported  earlier  in  Hendrickson  and  Collins  (1996), 
Taylor  and  Todd  (1995b)  used  structural  equation  modeling  to  account  for  obtained 
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results.  However,  instead  of  relying  on  subjective  estimates  of  IT  use  and  self  reports  of 
intended  use  (Davis,  1989;  Davis  et  al.,  1989;  Mathieson,  1991),  Taylor  and  Todd 
(1995b)  obtained  actual  usage  statistics  which  they  felt  would  provide  stronger  support 
for  TAM  than  subjective  usage  or  intentional  measures  collected  coincidentally  with 
other  self-report  measurements  of  TAM  variables.  Their  results  indicated  all  three 
models  provided  a  comparable  fit  to  the  obtained  data  for  explaining  variability  in  usage. 
However,  the  decomposed  TPB  did  a  slightly  better  job  of  explaining  behavioral 
intention  (in  direct  contradiction  to  the  findings  of  Mathieson,  1991)  than  either  the 
simple  TPB  or  TAM  (Taylor  and  Todd,  1995b:  166). 

Interpreting  these  results  takes  us  back  once  again  to  a  discussion  of  practicality. 
The  ability  of  the  decomposed  TPB  to  explain  more  variance  in  user  intentions  is  not  at 
all  that  surprising  given  the  fact  it  proposes  1 1  separate  determinants  for  behavior  and 
intention  as  opposed  to  TAM’s  three.  However,  when  considering  IT  implementation, 
the  variable  of  interest  is  often  usage  behavior  and  not  behavioral  intentions.  In  this 
respect,  it  is  not  clear  the  tradeoff  between  the  decomposed  TPB’s  slightly  higher 
predictive  power  of  intention  outweighs  the  parsimony  of  TAM  for  describing  the  same 
phenomena,  and  comparable  ability  to  explain  usage  behaviors  (Taylor  and  Todd, 

1995b:  169).  Thus,  despite  findings  which  ran  counter  to  Mathieson’s  (1991)  concerning 
TBP’s  predictive  power  for  intention,  Taylor  and  Todd’s  (1995b)  study  painted  virtually 
the  same  picture  for  TAM  as  did  Mathieson  (1991)  in  the  final  analysis:  “...if  the  central 
goal  is  to  predict  IT  usage,  it  can  be  argued  that  TAM  is  preferable.  However,  the 
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decomposed  TPB  model  provides  a  more  complete  understanding  of  the  determinants  of 
intention”  (Taylor  and  Todd,  1995b:  169). 

Extending  TAM  Through  User  Experience 

From  the  evidence  presented  thus  far,  TAM,  as  a  basic  paradigm  for 
understanding  IT  user  behaviors,  has  enjoyed  a  good  deal  of  empirical  support.  However, 
previous  findings  also  suggest  the  actual  mechanisms  by  which  user  behavior  is 
determined  are  more  complex  than  those  proposed  by  TAM  (Davis  et  al.,  1989; 
Mathieson,  1991;  Taylor  and  Todd,  1995b).  One  of  the  primary  variables  thought  to 
mitigate  the  predictive  and  explanatory  power  of  TAM  is  the  passage  of  time,  or,  user 
experience  with  the  IT  in  question.  For  example,  Adams  et  al.  (1992)  suggested  their 
findings  might  be  better  explained  if  perceptions  of  ease  of  use  varied  as  a  fiinction  of 
user  experience  level  (Adams  et  al.,  1992:243). 

In  an  attempt  to  quantify  the  magnitude  of  these  temporal  effects,  recent  research 
efforts  have  proposed  an  “augmented”  or  “revised”  TAM  which  specifically  addresses 
how  the  interrelationships  between  usefulness  and  ease  of  use,  attitudes,  intentions,  and 
user  behaviors  vary  as  a  function  of  time  or  experience.  One  such  study  conducted  by 
Taylor  and  Todd  (1995a)  was  actually  an  offshoot  of  their  earlier  work  comparing  TAM 
to  the  competing  TBP  models  (Taylor  and  Todd,  1995b).  The  same  usage  statistics  fi-om 
their  previous  study  were  reexamined,  factoring  in  the  experience  level  of  the  IT  users. 

Taylor  and  Todd  (1995a)  also  proposed  a  revised  TAM  which  incorporated  the 
normative  and  behavioral  control  aspects  of  TPB,  partially  due  to  the  results  obtained  by 
Mathieson  (1991)  and  Taylor  and  Todd  (1995b),  in  which  these  factors  were  observed  to 
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influence  behavior  (even  though  TAM  accounted  for  more  behavioral  variability  in  some 
cases).  Figure  7  illustrates  Taylor  and  Todd’s  (1995a)  augmented  TAM. 


From  calculations  of  overall  fit,  path  significance,  and  predictive  power  of  the 
proposed  model,  Taylor  and  Todd  (1995a)  fovmd  the  augmented  TAM  reasonably 
explained  usage  variability  for  both  experienced  and  inexperienced  IT  users.  Within  the 
relevant  variables  of  TAM’s  original  theoretical  fi'amework,  intention  was  found  to  be  a 
stronger  predictor  of  behavior  for  experienced  users,  while  usefulness  and  ease  of  use 
were  found  to  be  stronger  predictors  of  intention  and  attitude,  respectively,  for 
inexperienced  users.  Within  the  variables  “borrowed”  from  TBP  for  the  augmented 
TAM,  the  impact  of  subjective  norms  on  intent  were  not  found  to  differ  as  a  function  of 
experience,  although  the  behavioral  control  factor  did  prove  more  important  for 
inexperienced  users  in  determining  use  (Taylor  and  Todd,  1995a:565). 

Based  on  their  findings,  Taylor  and  Todd  (1995a)  concluded  that  the  knowledge 
gained  from  experience  with  IT  usage  creates  a  more  stable  intention-to-behavior 
relationship,  reducing  the  relative  effects  of  usefulness  and  ease  of  use  for  the  user; 
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however,  user  behaviors  for  the  inexperienced  are  more  strongly  influenced  by  these 
antecedents,  while  their  intentions  might  not  necessarily  translate  into  actual  use  (Taylor 
and  Todd,  1995a:563, 565).  From  an  academic  or  practical  standpoint,  these  results  did 
not  fully  discount  the  overall  usefulness  of  TAM  as  a  basic  theoretical  framework  for 
understanding  usage  behaviors,  but  they  did  underscore  the  apparently  large  and 
heretofore  imexamined  changes  which  occur  in  the  determinants  of  system  use  over  time 
(Taylor  and  Todd,  1995a:566). 

Szajna  (1996)  also  sought  to  empirically  examine  the  temporal  changes  in  TAM, 
specifically,  those  first  noted  by  Davis  et  al.  (1989)  in  which  the  direct  effect  of  ease  of 
use  on  intention  was  fully  mediated  by  usefulness  following  user  experience  with  IT 
systems  (Davis  et  al.,  1989:994).  Consequently  Szajna  (1996)  used  a  very  similar 
methodology  to  that  of  Davis  et  al.  (1989)  for  her  investigation,  including  the  timing  of 
ease  of  use,  usefulness,  and  intention  measurements  at  both  pre-  (IT  introduction)  and 
post-  (history  of  IT  systems  use)  implementation  stages  (Szajna,  1996:86). 

In  her  longitudinal  study  of  college  students’  use  of  an  electronic  mail  system, 
Szajna  (1996)  foxmd  once  users  became  more  experienced,  usefulness  not  only 
determined  behavioral  intention,  but  also  usage  behaviors.  In  addition,  ease  of  use,  while 
having  no  direct  effect  on  intention  or  usage  for  either  experienced  or  inexperienced 
users — ^implying  a  wholly  indirect  effect  consistent  with  Davis’  (1989)  earlier 
investigations — did  influence  perceptions  of  usefulness  more  strongly  for  inexperienced 
users  than  for  experienced  users  (Szajna,  1996:88).  These  findings  seem  intuitively  valid 
as  they  suggest  once  users  gain  experience  using  IT,  ease  of  use  becomes  less  important 
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in  determining  their  beliefs  about  how  useful  that  system  will  be  (Szajna,  1996:89).  In 
addition,  the  relationship  between  intention  and  behavior  was  much  stronger  for 
experienced  versus  inexperienced  users  (Szajna,  1996:90),  mirroring  results  reported  by 
Taylor  and  Todd  (1995a).  The  overall  pattern  of  Szajna’s  (1996)  results  strongly  suggest 
the  consideration  of  an  experience-related  component  within  the  original  TAM  for 
improved  applicability  across  IT  implementation  conditions. 

A  rather  important  aside  which  bears  repetition  at  this  point  is  the  predominantly 
strong  indication  that  self-reported  system  use  was  not  an  adequate  substitute  measure  for 
actual  system  use.  Szajna  (1996)  observed  little  correlation  between  self-reported  system 
use  and  objective  measures  of  system  use.  Furthermore,  the  strong  relationship  observed 
between  intentions  and  self  reported  use  was  discounted  by  the  weak  relationship 
observed  between  intentions  and  actual  IT  use  (Szajna,  1996:89).  Management 
implications  for  these  findings  suggest  careful  attention  must  be  paid  when  using  self- 
report  measurements  for  making  evaluations  of  IT  alternatives  or  the  relative  success  of 
delivered  IT  systems. 

Rather  than  applying  the  notion  of  temporal  changes  to  the  relationships  between 
the  TAM  variables,  Chau  (1996)  proposed  a  novel  variation  of  TAM  which  specifies 
temporally  based  changes  within  TAM  variables.  Specifically,  Chau  (1996)  proposed  a 
modified  TAM  in  which  usefulness  itself  is  divided  between  perceptions  of  long-term 
and  near-term  usefulness.  Long-  term  usefulness  was  defined  as  long-term  job-related 
benefits  of  having  knowledge  of  a  particular  technology  including  issues  associated  with 
cross-training,  job  security,  or  future  promotions.  Near-term  usefulness  referred  to 
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perceptions  of  the  task-related  benefits  of  using  a  particular  IT  system  and  was  most  akin 
to  the  traditional  notion  of  usefulness  as  defined  by  TAM  (Chau,  1996:189,  191).  Chau’s 
modified  TAM  appears  in  Figure  8. 


Short  Term 
Usefulness 

Ease  of  Use 

inicnuun 

Long  Term 
Usefulness 

Figure  8.  Modified  TAM  (Chau,  1996:191) 

Unlike  most  other  TAM  derivatives,  Chau’s  (1996)  model  did  not  include  measures  of 
usage  behavior,  direct  or  subjective;  the  hypothesized  link  between  intention  and  use 
(Davis,  1989;  Fishbein  and  Ajzen,  1975)  was  deemed  strong  enough  to  warrant 
investigation  only  the  point  of  the  determinant  of  usage  behavior,  rather  than  to  the 
behavior  itself  (Chau,  1996:190). 

In  his  study  of  alternative  software  selection  and  use  in  a  non-profit  organization, 
Chau’s  (1996)  results  had  similar  implications  for  the  relationship  between  ease  of  use, 
usefulness,  and  intention  as  those  reported  previously  by  Davis  (1989):  ease  of  use  had  a 
strong  effect  on  perceptions  of  (near-term)  usefulness,  but  no  direct  effect  on  intention  or 
long-term  usefulness  (Chau,  1996:197-198).  Hence,  as  the  user’s  fi-ame  of  reference 
moved  beyond  the  immediate  task  at  hand,  factors  pertaining  to  ease  of  use  became  less 
important  (Chau,  1996:201).  Perceptions  of  both  near-  and  long-term  usefulness  were 


35 


also  found  to  be  significant  factors  affecting  intention,  indicating  the  modified  TAM 
could  reasonably  account  for  variability  in  behavioral  intentions. 

Self  Efficacy  Theory 

Stepping  outside  the  boimdaries  of  IT-specific  thought  for  a  moment,  one  other 
theory  of  relevance  to  the  current  discussion  has  yet  to  be  explored:  self  efficacy  theory. 
According  to  Bandura  (1986),  self  efficacy  is  defined  as  “people’s  judgments  of  their 
capabilities  to  organize  and  execute  course  of  action  required  to  attain  designated  types  of 
performances”  (Bandura,  1986:391).  In  lay  terms,  self  efficacy  is  commonly  understood 
as  a  person’s  beliefs  about  his  or  her  abilities  to  accomplish  a  particular  task  or  attain 
some  desired  level  of  performance. 

What  is  important  about  self-efficacy  to  the  discussion  of  IT  use  is  the  influence 
of  self-efficacy  beliefs  on  behavior.  Perceptions  of  self-efficacy  have  been  shown  to 
influence  thought  patterns,  actions,  and  emotional  arousal  (Bandura,  1982:122).  More 
specifically,  self-efficacy  theory  indicates  “people  tend  to  avoid  tasks  and  situations  they 
believe  exceed  their  capabilities,  but  they  imdertake  and  perform  assuredly  activities  they 
judge  themselves  capable  of  handling”  (Bandura,  1986:393). 

By  applying  this  theory  to  the  case  of  IT  use,  it  is  suggested  that  any  aspect  of  the 
IT  usage  experience  which  raises  or  lowers  self-efficacy  beliefs  has  the  potential  to 
influence  subsequent  usage  behaviors.  Thus,  people  who  believe  they  are  capable  of 
using  IT  to  accomplish  their  tasks  are  much  more  likely  to  use  IT  than  those  who  do  not 
share  similar  self-efficacy  beliefs.  But  where  do  these  self-efficacy  beliefs  come  from — 
and  how  do  they  relate  to  our  discussion  of  experience  and  temporal  factors  in  IT  use? 
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Fortunately,  self-efficacy  theory  nicely  answers  these  questions.  According  to 
Bandura  (1986),  performance  attainment  (i.e.  direct  experience)  “provide[s]  the  most 
influential  source  of  efficacy  information  because  it  is  based  on  authentic  mastery 
experiences.  Successes  raise  efficacy  appraisals;  repeated  failures  lower  them,  especially 
if  the  failures  occur  early  in  the  course  of  events  and  do  not  reflect  lack  of  effort  or 
adverse  external  circumstances”  (Bandura,  1986:399). 

The  ramifications  self-efficacy  theory  have  for  questions  concerning  IT  use 
should  be  clear.  If  IT-related  self-efficacy  beliefs  are  developed  from  successful  or  failed 
attempts  at  using  IT,  then  IT-related  self-efficacy  beliefs  must,  by  definition,  be 
determined  by  users’  experiences  with  IT.  Furthermore,  if  the  successes  or  failures  we 
have  when  using  IT  are  the  most  influential  sources  of  self-efficacy  beliefs  (as  Bandura, 
1986,  maintains),  then  those  factors  which  most  directly  influence  the  repeated  success  of 
a  user’s  IT  experiences  carry  the  most  potential  for  determining  or  influencing  future 
usage  behaviors.  Therefore,  it  is  reasonable  to  assume  that  the  rash  of  apparent 
temporally  related  results  reported  in  the  studies  above  may  have  been  determined  in  part 
by  the  formation  of  self-efficacy  beliefs  over  the  course  of  the  users’  IT  experiences. 
Synthesis 

As  a  conceptual  framework  for  gaining  a  basic  understanding  of  IT  user  behavior, 
TAM  has  withstood  the  test  of  time  and  empirical  scrutiny  for  nearly  the  past  decade.  In 
general,  the  following  properties  of  TAM  have  been  supported  by  available  data: 

1)  Both  ease  of  use  and  usefulness  appear  to  be  relevant  for  determining  IT  use. 

2)  Ease  of  use  is  likely  to  be  of  secondary  concern  to  issues  of  overall  usefulness. 


3)  The  causal  mechanisms  of  IT  use  most  likely  follow  a  path  from  belief  to 
attitude  to  intention  to  behavior. 

It  has  also  been  suggested  that  additional  factors  may  extend  TAM’s  ability  to 
account  for  user  behavior.  While  a  variety  of  such  factors  have  been  introduced 
throughout  the  relevant  literature  (behavioral  control,  subjective  norms,  nature  of  the  IT 
system  itself,  etc.),  most  researchers  have  acknowledged  (or  at  least  observed)  the  effects 
of  user  experience  in  determining  the  magnitude  or  direction  of  the  relationships  between 
TAM  factors.  This  experience  component  is  further  suggested  and  supported  by  the  basic 
postulates  of  self-efficacy  theory,  a  robust  and  empirically  well-supported  paradigm  for 
understanding  cognitive  and  behavioral  determinants  in  its  own  right. 

It  is  from  this  point  the  present  investigation  begins.  Although  user  experience 
level  was  foimd  to  be  a  significant  factor  in  many  investigations  of  TAM,  it  is  reasonable 
to  assume  that  more  than  just  experience  per  se  influences  the  ease  of  use-usefulness- 
usage  relationship.  In  fact,  self-efficacy  theory  suggests  something  more  than  simply  an 
increased  familiarity  with  system  features  or  how  the  system  works  is  at  work  here — 
something  which  tangibly  and  directly  affects  the  relative  success  people  have  when 
using  IT.  As  suggested  earlier,  these  factors  may  well  be  responsible  for  some  of  the 
temporally  dependent  results  reported  above  (Adams  et  al.,  1992;  Chau,  1996;  Davis  et 
al.,  1989;  Szajna,  1996;  Taylor  and  Todd,  1995a). 

Returning  to  TRA,  the  theoretical  backdrop  upon  which  TAM  was  conceived,  it 
was  proposed  that  people  use  knowledge  gained  from  prior  experience  to  form  their 
intentions  (Fishbein  and  Ajzen,  1975:332;  Taylor  and  Todd,  1995a:565).  Indeed,  the 
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notion  of  feedback  from  behavior  to  belief  was  incorporated  into  the  original  conceptual 
framework  for  TRA  (although  the  formal  model  is  often  depicted  without  it  in  the  IT 
literature).  Perhaps  there  is  something  about  the  nature  of  this  feedback,  or  the 
information  gained  from  the  relative  success  of  the  usage  behaviors  enjoyed  during  user 
experiences,  which  better  accounts  for  subsequent  IT  use  (other  than  merely  length  of 
time  spent  engaged  in  usage  behaviors). 

Hopefully,  the  present  investigation  will  bring  to  light  some  of  the  relevant  issues 
associated  with  user  experience  which  will,  in  turn,  “feed  back”  on  the  determinants  of 
future  IT  systems  use.  Quality  was  chosen  as  a  possible  construct  of  interest  because  it 
not  only  conveys  beliefs  about  the  interactions  between  user  and  IT  (i.e.  in  order  to  have  a 
“high”  or  “low”  quality  interaction  with  information  technology,  X,  Y,  and  Z  must  be 
true),  but  also  evaluative  information  about  the  nature  of  those  interactions  themselves 
(i.e.  the  experience  of  the  IT  interaction  was  of  “high”  or  “low”  quality).  It  is  further 
hoped  the  results  will  suggest  in  what  ways  usage  experience  mediates  the  relationships 
between  the  antecedents  of  IT  use  and  use  itself 
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III.  Methodology 


Pilot  Studv/Item  Generation 

Data  for  this  study  were  collected  using  a  two  tiered  approach.  Initial  efforts 
followed  along  the  lines  of  a  modified  Critical  Incident  Technique  (CIT)  (Flanagan, 
1954).  CIT  was  originally  conceived  of  as  an  objective  means  of  gathering  “certain 
important  facts  concerning  behavior  in  defined  situations”  (Flanagan,  1954:335). 

The  CIT  method  itself  is  quite  simple:  subjects  are  asked  a  series  of  open-ended 
questions  in  which  they  focus  on  the  “best”  and  “worst”  exemplars  of  a  particular 
behavior  (i.e.  a  “critical  incident”).  The  aim  of  this  line  of  questioning  is  to  determine  the 
common  antecedents  to  specific  judgments  regarding  exhibited  behaviors — ^what  made 
for  the  “best”  or  “worst”  bombardier,  contracting  officer,  or  pilot,  for  example.  Although 
the  present  study  focuses  on  judgments  of  experience  rather  than  judgments  of  behavior, 
there  were  no  theoretical  reasons  to  assume  the  CIT  would  not  be  an  equally  effective 
methodology  for  gathering  important  facts  concerning  circumstances  or  issues  associated 
with  defined  situations — ^namely,  circumstances  or  issues  associated  with  the  “quality”  of 
Internet  experiences. 

Following  the  CIT’s  general  methodology,  the  first  step  was  to  identify  those  key 
circumstances  or  issues  which  Internet  users  associate  with  the  “best”  (high)  and  “worst” 
(low)  quality  of  experience.  Because  the  aim  of  this  study  was  to  identify  and  fully 
explore  a  factor  hypothesized  to  moderate  the  attitude-intention-behavior  relationship 
between  IT  and  IT  user,  the  hope  was  to  generate  a  diverse  and  potentially  inclusive  set  of 
items  for  subsequent  examination.  To  this  end,  a  pool  of  heterogeneous  subjects  was 
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sought  for  the  first  portion  of  the  study — a  subject  pool  exhibiting  wide  variability  in 
terms  of  Internet  experience  (both  in  length  and  frequency),  means  of  access,  and  reasons 
for  use.  By  capitalizing  on  such  variability,  as  well  as  the  differences  in  perceptions  of 
experiential  quality  rmdoubtedly  influenced  by  both  successful  and  non-successfiil 
Internet  experiences,  it  was  assumed  the  variety  of  motivations  and  experience  levels 
would  provide  reasonable  assurance  of  requisite  variety  within  the  pilot  responses. 

Bearing  these  assumptions  in  mind,  a  convenience  sample  of  23  subjects  was 
identified  to  participate  during  the  item  generation  phase.  Eighteen  subjects  were  AFIT 
master’s  candidates  in  the  Information  Resources  Management  (IRM)  program,  one  was 
a  non- AFIT  Air  Force  officer,  and  four  were  non-DoD  civilians.  A  general,  open-ended 
questionnaire  was  sent  to  each  subject  via  electronic  mail  with  an  83  percent  return  rate. 
Subject  age  ranged  from  24  to  55;  reported  length  of  Internet  usage,  as  shown  in  Table  1, 
indicated  a  relatively  well-balanced  mix  of  experienced  and  inexperienced  Internet  users. 


Table  1.  Experience  Level  of  Pilot  Study  Subjects 


Number  of  Respondents 

Experience  Level 

2 

Less  than  6  months 

3 

6  months  to  1  year 

5 

1  to  2  years 

5 

2  to  3  years 

3 

More  than  3  years 

Only  three  questions  were  posed  in  the  questionnaire: 

1)  If  you  were  to  characterize  your  overall  experience  with  the 
Internet  as  high  quality  (whatever  that  means  to  you  personally), 
what  factors  would  contribute,  or  do  you  think  would  contribute,  to 
that  characterization? 
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2)  If  you  were  to  characterize  your  overall  experience  v^ith  the 
Internet  as  low  quality  (whatever  that  means  to  you  personally), 
what  factors  would  contribute,  or  do  you  think  would  contribute,  to 
that  characterization? 

3)  For  what  activities/reasons  (specific)  or  categories  of 
activities/reasons  (general),  do  you  use  the  Internet  (personal  or 
work-related)? 

Subjects  were  asked  to  answer  each  question  as  thoughtfully  as  possible,  and  to  make 
suppositions  concerning  either  of  the  first  two  questions  if  they  could  not  honestly 
characterize  their  Internet  experiences  as  being  of  either  high  or  low  quality. 

Instrument  Development 

Although  93  separate  responses  were  collected  during  the  initial  phase  of  this 
study,  overt  commonalties  between  subject  responses  reduced  this  potential  pool  to  only 
28  unique  response  items.  Frequency  of  response  was  used  as  a  surrogate  measure  of 
item  “importance”  to  a  common  perception  of  quality  of  experience  (common  across 
subjects);  therefore,  any  single  item  not  having  a  response  frequency  of  at  least  two  was 
also  excluded  from  further  consideration.  In  addition,  the  exploratory  nature  of  this  study 
allowed  for  a  small  degree  of  creative  license  when  deciding  which  items  seemed  too 
confusing  (or  could  not  easily  be  explained  within  the  confines  of  a  short  questionnaire) 
to  be  included  in  the  final  instrument,  or  which  items  warranted  further  elaboration.  For 
example,  “ease  of  use”  was  commonly  cited  as  a  determinant  for  high  quality  of 
experience.  At  any  one  time,  however,  Internet  usage  is,  by  design,  subject  to  both  the 
Web  browser  interface  and  the  interface  created  at  the  Web-s/Ye  itself  Consequently, 


42 


ease  of  use  was  broken  out  between  separate  items  for  browsers  and  Web-sites  to  capture 
any  such  differences  which  might  appreciably  affect  perceptions  of  quality  of  experience. 

One  unforeseen  difficulty  in  developing  a  measurement  instrument  from  the 
initial  responses  was  the  evaluative  nature  of  the  response  items  themselves.  Because  the 
CIT  generates  items  corresponding  to  “best”  or  “worst”  case  scenarios,  response  items 
often  carried  these  evaluative  qualities  with  them.  For  example,  “fast  response”  was 
frequently  cited  as  a  determinant  of  high  quality  of  experience,  while  “slow  access”  was 
equally  common  for  determining  low  quality  of  experience — ^yet  it  is  reasonable  to 
assume  both  refer  to  the  speed  at  which  users  access  the  Internet.  However,  asking  a  user 
if  slow  access  contributed  to  perceptions  of  high  quality  makes  little  sense  and  could 
easily  confuse  the  subject  as  to  the  point  of  the  question. 

Consequently,  any  evaluative  connotations  in  response  item  content  were 
reconciled  (with  varying  degrees  of  success)  through  corresponding  “evaluation-neutral” 
items.  For  example,  “advertising  clutter,”  commonly  cited  as  a  determinant  of  low 
quality  of  experience,  was  replaced  with  “advertising  presence.”  This  allowed  for  the 
possibility  that  while  some  might  find  Internet  advertising  useful,  others  might  consider  it 
a  nuisance — ^but  everyone  would  be  free  to  choose  the  same  survey  item  to  refer  to 
different  levels  of  quality  of  experience  without  being  encumbered  by  any  subjective 
evaluation  implicit  in  the  item  itself. 

Another  problem  which  quickly  became  self-evident  was  that  there  was  no 
theoretical  justification  for  assuming  the  absence  of  a  high  quality  experiential  attribute 
automatically  meant  the  presence  of  the  “opposite”  low  quality  attribute.  In  fact,  it  was 


43 


very  possible  the  same  item  could  be  extremely  important  for  determining  high  quality  of 
experience,  but  have  an  entirely  different,  yet  not  opposite,  effect  on  determining  low 
quality  of  experience.  Therefore,  one  working  assumption  was  that  the  factors  which 
determine  high  quality  of  experience  were  independent  of  the  factors  determining  low 
quality  of  experience.  Given  this  assumption,  treating  high  quality  of  experience  and  low 
quality  experience  as  separate  entities  on  the  measurement  instrument  seemed  the  more 
conservative  approach  given  the  exploratory  nature  of  this  research;  certainly  less  tenuous 
than  trying  to  develop  and  defend  a  scale  producing  “composite”  quality  scores  using 
items  derived  from  determinants  of  both  high  and  low  quality  of  experience. 

After  consideration  of  the  issues  above,  a  final  list  of  1 9  separate  items  was 
prepared  and  used  for  the  next  phase  of  the  study.  To  determine  how  important  these 
items  were  to  user  perceptions  of  quality  of  experience,  each  item  was  rated  individually 
in  terms  of  relative  importance  to  the  subject’s  overall  perceptions  of  high  and  low 
quality  of  experience.  Each  item  was  rated  using  a  7-point,  Likert-type  scale,  ranging  in 
degree  between  the  anchors  of  “Extremely  Important”  and  “Not  at  all  Important.” 

Again,  because  of  the  assumption  that  factors  which  influence  high  quality  of 
experience  were  not  directly  dependent  upon  factors  which  influence  low  quality  of 
experience,  both  cases  of  high  and  low  quality  of  experience  were  addressed  on  the  same 
survey  form.  To  reduce  the  likelihood  that  subjects’  scores  on  items  related  to  high 
quality  influenced  their  scores  on  the  same  items  which  related  to  low  quality,  each  item 
appeared  twice,  once  on  the  first  side  concerning  high  quality  of  experience,  and  once  on 
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the  other  in  a  different,  random-order  location,  concerning  low  quality  of  experience. 

The  questionnaire  itself  appears  in  Appendix  A. 

Instrument  Administration 

Given  the  Air  Force’s  reputation  as  a  technologically  advanced  service,  there  were 
no  theoretical  reasons  to  assume  perceptions  regarding  the  Internet,  or  information 
technology  in  general,  would  differ  systematically  between  subjects  based  upon  factors 
such  as  rank  or  MAJCOM.  On  the  contrary,  the  relative  educational,  cultural,  and 
attitudinal  similarities  between  Air  Force  personnel  might  conceivably  offer  some 
assurance  of  consistency  between  subject  responses.  In  addition,  the  pilot  research  was 
limited  to  a  very  narrow  segment  of  the  Air  Force  population,  primarily  AFIT IRM 
smdents;  a  less  specialized  sample  was  desired  to  increase  the  degree  to  which  results 
could  be  generalized  to  the  population  at  large. 

As  expressed  in  earlier  sections  of  this  report,  one  of  the  goals  of  this  exploration 
was  to  examine  the  possibility  of  any  imderlying  traits  or  factors  which  together  comprise 
a  general  notion  of  quality  of  experience.  To  achieve  this  goal,  the  original  intent  was  to 
examine  the  patterns  of  item  responses  subject  via  factor  analysis — the  results  of  which 
would  guide  and  direct  further  exploration  towards  a  simple  and  elegant  factor  structure. 
One  rule  of  thumb  suggests  the  bare  minimum  number  of  subjects  for  a  reasonable  factor 
analysis  (provided  there  are  at  least  100  total  subjects)  is  5  per  factor/variable,  or  10 
subjects  per  measurement  scale  item,  whichever  is  greater  (Streiner,  1994:140). 

However,  due  to  the  exploratory  nature  of  this  investigation,  there  were  no  theoretical 
assumptions  concerning  the  number  of  factors  which  might  be  expected;  hence,  no 
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reasonable  means  by  which  to  estimate  an  appropriate  sample  size.  Given  the  practical 
time  limitations  and  resource  constraints,  provisions  were  finally  made  to  expect, 
accommodate,  and  analyze  roughly  200  to  300  usable  survey  responses,  with  the 
understanding  that  the  number  of  responses  received  would  ultimately  dictate  the  extent 
to  which  factor  analysis,  or  any  other  statistical  procedure,  could  be  applied  to  the  data. 
Subjects 

Based  on  an  expected  return  rate  of  roughly  20  percent,  a  random  sampling  of 
1600  Air  Force  officers  and  enlisted  personnel  was  identified  to  receive  the  questionnaire 
via  direct  mail.  Unfortunately,  only  148  usable  replies  were  received,  for  a  return  rate  of 
only  9.25  percent.  An  additional  40  surveys  were  for  lack  of  prior  Internet  experience. 
Subject  demographics  indicated  respondents  came  from  a  wide  range  of  MAJCOMs,  with 
nearly  twice  the  number  of  officers  as  enlisted  (98  to  50,  respectively).  Subject  age 
ranged  from  18  to  55,  with  a  mean  of  34.4  years.  Experience  level  was  measured  via  one 
of  six  categories:  no  experience,  0-6  months,  6  months- 1  year,  1-2  years,  2-3  years,  and 
over  3  years.  The  distribution  of  experience  level  is  given  in  Figure  9. 


Final  survey  responses  were  tallied  using  Microsoft  Excel  for  Windows  95,  and  analyzed 
using  the  statistical  analysis  software  package  SPSS  for  Windows,  Version  7.0. 

Summary 

The  following  is  a  brief  synopsis  of  the  methodological  activities  used  during  this 
study.  First,  potential  survey  items  were  gathered  using  the  CIT  from  a  small  sample  of 
Internet  users  at  varying  levels  of  experience.  Items  were  then  subjectively  analyzed  for 
commonalties  and  evaluative  content,  and  finally  pared  down  to  a  list  of  19  seemingly 
important  characteristics  which  pertained  to  perceptions  of  quality  of  experience.  A 
Likert-type  scale  was  chosen  to  isolate  the  degree  to  which  each  item  contributed  to  those 
overall  perceptions  of  quality.  Further  considerations  of  the  potential  differences  between 
the  determinants  of  high  and  low  quality  of  experience  prompted  the  inclusion  of  both  on 
the  survey  as  constructs  of  interest.  Finally,  surveys  were  mailed  to  a  random  sampling 
of  Air  Force  personnel  and  returns  based  on  voluntary  participation.  Details  of  the 
obtained  results  and  subsequent  analysis  appear  in  the  next  chapters. 


IV.  Results  and  Analysis 


Assessments  of  Scale  Internal  Consistency 

As  indicated  in  the  previous  chapter,  there  were  no  theoretical  justifications  for 
assuming  the  determinants  of  high  quality  of  experience  carried  the  same  relative 
“weight”  for  determining  low  quality  of  experience.  Consequently,  each  set  of  survey 
items  (i.e.  “high”  and  “low”  quality)  was  treated  as  a  separate  measurement  scale  for  the 
initial  phases  of  analysis  (or  until  the  results  suggested  otherwise).  My  expectation  was 
that  internal  consistency  might  increase  as  less  relevant  items  were  successively  dropped. 
Cronbach’s  alpha  for  the  high  quality  of  experience  scale  was  0.87;  alpha  for  the  low 
quality  of  experience  scale  was  0.92.  In  addition,  estimates  of  internal  consistency  for 
high  and  low  quality  of  experience  could  be  increased  to  0.88  and  0.94,  respectively,  with 
the  omission  of  item  19,  “Advertising  Presence.”  However,  given  the  unexplored 
territory  the  quality  of  experience  construct  represented,  it  seemed  prudent  at  this  juncture 
to  take  a  somewhat  conservative  approach  when  interpreting  the  results.  Pursuant  to 
these  objectives,  item  19  was  retained  for  further  consideration  pending  the  outcome  of 
subsequent  analyses. 

Interpreting  Obtained  Results 

Given  the  relatively  unstructured  nature  of  the  instrument  development  activities, 
both  estimates  of  internal  consistency  were  surprisingly  high.  The  high  internal 
consistency  estimates  also  suggest  that  responses  to  the  scales  themselves  were  not 
dominated  by  random  error.  However,  this  begged  the  question  which  was  at  the  heart  of 
the  current  exploratory  study — ^what  exactly  were  these  scales  measuring? 


48 


This  question  accentuates  the  need  for  additional  study  of  the  quality  of 
experience  construct  and  will  be  addressed  in  greater  detail  in  the  following  chapter.  At 
this  time,  it  is  important  simply  to  recognize  that  the  purpose  of  the  current  study  was  to 
investigate  the  components  of  a  construct  which,  based  upon  relevant  behavioral  science 
and  IT-related  research  findings,  should  (theoretically)  influence  IT  user  behaviors. 
However,  no  measure  of  IT  user  behavior  was  gathered  in  conjunction  with  this  study, 
nor  was  the  study  designed  to  accommodate  such  a  measure.  Within  the  context  of  initial 
construct  exploration  and  survey  development,  the  scope  of  this  research  effort  included 
simply  establishing  the  foimdation  and  defining  the  boundaries  of  whatever  notion  IT 
users  associated  with  the  terms  “quality  of  experience,”  or  examining  if  such  a  notion  was 
consistently  held  at  all. 

Turning  attention  back  to  the  results  reported  thus  far,  it  seems  the  investigative 
objectives  discussed  above  were  merited.  The  high  internal  consistency  estimates  for 
both  measurement  scales  certainly  suggested  something  consistent  and  systematic  was 
happening  during  the  usage  experiences  of  the  sampled  Internet  users — something  which 
was  captured  within  the  survey  items.  Therefore,  the  remainder  of  the  analysis  focused 
on  refining  the  quality  of  experience  measurement  scales  themselves,  and  quantifying  the 
emerging  relationships  between  survey  items  to  the  greatest  degree  practicable. 

Factor  Structure  within  the  Measurement  Scales 

As  indicated  in  the  section  above,  the  current  investigation  was  not  designed  to 
relate  the  scores  of  any  emergent  factors  to  other  IT-related  measurements  (e.g.  to 
measures  of  actual  use).  Nevertheless,  there  still  existed  the  very  real  possibility  that 
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certain  survey  items  would  cluster  together— this  in  itself  could  provide  some  insight  into 
the  makeup  of  a  quality-related  construct.  In  addition,  an  intuitive  line  of  reasoning 
suggested  the  likelihood  of  finding  a  pattern  of  responses  on  a  quality-related  survey 
attributable  to  a  smaller  number  of  underlying  traits  or  factors  because  the  term  “quality” 
itself  can  convey  so  many  different  meanings  in  so  many  different  contexts. 

Principle  axis  factoring  was  used  to  extract  an  initial  set  of  potential  factors  from 
the  obtained  data.  A  factor  matrix  for  the  high  quality  of  experience  scale  items  is 
presented  in  Table  2. 


Table  2.  Initial  Item-Factor  Loadings  for  High  Quality  of  Experience  Scale  Data 


Item 

Factor  1 

Factor  2 

Factor  3 

Factor  4 

Factor  5 

HQ13 

.74785 

-.24770 

-.20404 

HQ9 

.69039 

-.59388 

.18180 

HQ12 

.67478 

-.26689 

HQ5 

.65589 

.40178 

.18812 

HQ8 

.64763 

.20948 

HQ15 

.62726 

-.15034 

.26888 

HQ14 

.62126 

.17635 

-.16453 

HQIO 

.61586 

-.16389 

-.38884 

HQ18 

.59444 

-.42276 

HQ17 

.57048 

-.22733 

HQll 

.53954 

.26344 

-.20141 

-.36822 

HQ16 

.53592 

-.16291 

-.24090 

HQ4 

.53221 

.49541 

.22817 

HQ6 

.45081 

.28797 

HQ3 

.38214 

.19138 

.37377 

HQ2 

.33578 

.61560 

.26974 

HQl 

.19754 

.58675 

-.30957 

.28579 

HQ19 

.31467 

-.53261 

HQ7 

.42686 

.44969 

Although  the  five-factor  structure  indicated  above  appears  somewhat  chaotic,  factor  1 
actually  accounted  for  over  30  percent  of  the  total  variance  in  item  responses;  the  next 
highest  factor  only  accounted  for  7.3  percent  of  the  total  variance  (a  ratio  exceeding  4:1 
for  the  first  two  eigenvalues).  The  fact  that  all  items  had  sizable  loads  on  factor  1, 
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coupled  with  the  pronounced  break  in  the  scree  plot  between  factor  1  and  the  remaining 
factors  (see  Figure  10),  suggested  that  the  high  quality  scale  was  roughly  unidimensional. 


Factor  Scree  Pbt 


Factor  Nunber 

Figure  10.  Factor  Scree  Plot  for  High  Quality  of  Experience  Scale  Data 

However,  there  is  a  distinct  possibility  that  this  apparent  unidimensionality  was  only 
artifactual;  lack  of  a  concrete  stimulus  may  have  encouraged  acquiescence,  while  the 
lack  of  multiple  items  for  some  potential  facets  of  quality  may  have  made  indications  of 
multiple  factors  unlikely — ^please  see  the  section  immediately  following  this  discussion 
for  more  thorough  consideration  of  these  possibilities. 

The  same  procedure  as  discussed  above  for  the  high  quality  of  experience  scale 
was  used  to  examine  the  data  obtained  from  the  low  quality  of  experience  scale;  the 
initial  factor  matrix  is  given  in  Table  3. 


51 


Table  3.  Initial  Item-Factor  Loadings  for  Low  Quality  of  Experience  Scale  Data 


Item 

Factor  1 

Factor  2 

Factor  3 

Factor  4 

LQ9 

.78749 

-.20781 

LQ14 

.77976 

-.15430 

LQIO 

.76956 

-.16401 

LQ8 

.75122 

-.21106 

LQ15 

.72604 

.27007 

LQ3 

.70497 

-.39404 

LQl 

.70151 

-.45862 

LQll 

.69832 

-.17593 

LQl  2 

.67461 

.27370 

.22837 

LQ17 

.67026 

.36021 

-.22511 

LQ6 

.65014 

-.39820 

.29342 

LQl  8 

.64593 

.22074 

.19719 

.32819 

LQ7 

.64470 

-.18985 

LQ4 

.63927 

.44182 

LQl  3 

.63511 

.24750 

.29795 

LQ2 

.61636 

-.36934 

LQl  6 

.59308 

.22530 

-.35265 

LQ5 

.54403 

.32937 

-.15057 

LQl  9 

.29988 

.17278 

Here  again,  factor  1  accounted  for  a  lion’s  share  of  the  total  response  variance:  over  44 
percent  this  time;  the  next  highest  factor  only  accounted  for  6  percent  (a  ratio  exceeding 
7: 1  for  the  first  two  eigenvalues).  The  resultant  scree  plot  for  the  low  quality  of 
experience  items  is  shown  in  Figure  11. 

Factor  Scree  Pbt 


Factor  Nunier 

Figure  11.  Factor  Scree  Plot  for  High  Quality  of  Experience  Scale  Data 
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The  pronounced  break  in  the  scree  plot  between  factor  1  and  the  other  factors,  and  the 
observation  that  all  survey  items  (with  the  exception  of  one)  loaded  strongly  with  this 
factor — ^in  fact,  all  the  items  as  a  whole  tended  to  load  more  strongly  with  the  single 
factor  than  they  did  in  the  case  of  the  high  quality  of  experience  data — suggested  this 
scale  was  unidimensional  as  well.  Again,  however,  one  of  several  potential  artifacts 
could  also  account  for  these  findings;  these  are  discussed  at  length  in  the  next  section. 

One  interesting  aspect  of  these  results  was  that  item  19,  “Advertising  Presence,” 
was  not  found  to  cluster  together  with  all  the  other  survey  items  on  the  low  quality  of 
experience  scale  as  it  had  on  the  high  quality  of  experience  scale  (although  the  item- 
factor  loading  in  that  case  was  somewhat  moderate:  0.3 1467).  However,  the  entire  set  of 
low  quality  of  experience  survey  items,  save  for  item  19,  still  comprised  a  single  factor. 
Considered  in  conjunction  with  the  results  of  the  internal  consistency  analyses  reported 
earlier  (indicating  both  measurement  scales  would  be  more  internally  consistent  without 
item  1 9),  the  factor  analyses  cast  additional  doubt  on  the  efficacy  of  the  “Advertising 
Presence”  item  for  assessing  overall  perceptions  of  quality  of  experience.  Consequently, 
any  revised  or  “final”  survey  form  produced  as  a  result  of  this  investigation  would  likely 
not  include  “Advertising  Presence”  for  its  lack  of  consistency  or  poor  predictive  ability  of 
total  item  responses  (relative  to  the  other  survey  items). 

Proceeding  Carefully  from  the  Results  of  Factor  Analysis 

Several  cautionary  notes  are  warranted  at  this  stage  of  the  discussion,  ones  which 
are  important  enough  to  be  echoed  again  during  the  analysis  in  the  following  chapter. 
First,  although  the  results  of  the  factor  analysis  seem  promising  on  the  surface,  the  reader 
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is  advised  to  keep  a  very  critical  eye  open  when  considering  the  implications  of  these 
results.  The  obtained  n  was  only  148,  well  below  the  desired  sample  size  of  10  subjects 
per  item  (as  suggested  by  Streiner,  1994).  Thus,  while  the  results  hint  at  the  possibility  of 
construct  unidimensionality,  they  may  just  as  easily  have  been  an  artifact  of  the  small 
sample  size.  Additional  sample  size  considerations  are  addressed  in  Chapter  V. 

Second,  and  perhaps  more  importantly,  the  obtained  factor  loadings  may  have 
been  artifacts  of  more  fundamental  issues  associated  with  the  measurement  task  itself. 
Because  the  survey  asked  for  ratings  of  individual  item  importance  subject  to  the  users’ 
entire  history  of  Internet  experiences,  there  was  no  concrete  stimulus  condition  upon 
which  to  base  those  estimates  of  relative  importance.  In  this  context,  a  concrete  stimulus 
condition  might  refer  to  the  user’s  last  significant  Internet  experience,  perhaps  more  than 
20  minutes  of  consecutive  usage,  or  the  user’s  most  recent  experience  with  a  particular 
Web  site  or  interface  design.  In  this  study,  however,  the  users  were  asked  to  relate  each 
survey  item  to  a  rather  vague  and  nebulous  notion  of  Internet  experience  in  general.  It  is 
very  possible  subjects  simply  acquiesced  to  the  demands  of  this  situation  by  answering 
indiscriminately  on  all  survey  items.  This  subject  vrill  be  revisited  in  Chapter  V,  along 
with  potential  remedies  for  the  future  study  of  quality  of  experience. 

Finally,  it  is  entirely  possible  that  factor  analysis  efforts  were  not  given  the 
appropriate  “chance”  to  find  an  underlying  factor  structure  within  the  context  of  this 
particular  measurement  instrument.  Factor  analysis  itself  could  try  to  extract  the 
underlying  “facets”  of  a  somewhat  multi-faceted  construct,  with  the  assumption  that 
multiple  items  are  associated  vrith  each  facet.  If  such  facets  actually  exist  in  this 
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measurement  instrument,  they  may  have  had  only  one  or  two  associated  survey  items. 
Thus,  the  apparent  scale  unidimensionality,  suggested  by  the  item-factor  loadings  and 
scree  plots  reported  in  the  previous  section,  could  be  accoimted  for  by  highly 
intercorrelated  and  internally  unidimensional  sub-scales  of  experiential  quality — sub¬ 
scales  which  simply  did  not  have  enough  individual  items  to  produce  separate  and 
discernible  factors.  This  eventuality  will  also  be  revisited  in  Chapter  V,  with  particular 
attention  paid  to  corrective  measures  suited  for  follow-on  research. 

Testing  the  Assumption  of  Independent  Measures 

Recall  one  of  the  key  working  assumptions  of  this  study  stipulated  that  the  factors 
which  influence  perceptions  of  high  quality  of  experience  are  not  necessarily  dependent 
on  those  which  influence  perceptions  of  low  quality.  Consequently,  survey  items  relating 
to  quality  of  experience  should  produce  response  data  independent  of  the  other  survey 
items.  Should  the  results  indicate  otherwise,  there  would  be  reason  to  suspect  the  factors 
which  influence  perceptions  of  high  and  low  quality  are  not  all  that  dissimilar.  Therefore, 
the  observed  relationships  between  scores  on  high  quality  items  to  scores  on  the  same 
low  quality  items  are  of  particular  consequence  to  the  current  investigation. 

An  inter-item  correlation  analysis  was  conducted  to  test  the  degree  to  which  the 
assumption  regarding  independent  measures  was  valid.  Assuming  reported  item  scores 
pertaining  to  high  quality  of  experience  actually  are  independent  of  low  quality  scores,  or 
that  the  determinants  of  high  quality  of  experience  do  not  carry  the  same  relative 
importance  as  the  determinants  of  low  quality  of  experience,  there  should  be  relatively 
little  significant  and  systematic  correlation  between  matched  high  and  low  quality  item 
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scores.  An  excerpt  from  the  inter-item  correlation  matrix  given  at  Appendix  B  appears 
below  in  Table  4. 


Table  4.  Inter-item  Correlations  Between  High  and  Low  Quality  of  Experience  Items 


Item  Pair 

Correlation 

Item  Pair 

Correlation 

HQ  1  - LQ  1 

0.363 

HQ  11 -LQ  11 

0.393 

HQ  2  - LQ  2 

0.506 

HQ  12 -LQ  12 

0.530 

HQ  3  - LQ  3 

0.417 

HQ  13 -LQ  13 

0.521 

HQ  4  - LQ  4 

0.469 

HQ14-LQ  14 

0.355 

HQ  5  - LQ  5 

0.506 

HQ  15 -LQ  15 

0.605 

HQ  6  - LQ  6 

0.413 

HQ  16 -LQ  16 

0.556 

HQ  7  - LQ  7 

0.548 

HQ  17 -LQ  17 

0.562 

HQ  8  - LQ  8 

0.309 

HQ  18 -LQ  18 

0.538 

HQ  9  - LQ  9 

0.402 

HQ  19 -LQ  19 

0.225 

HQ  10 -LQ  10 

0.400 

**  All  correlation  coefficients  sig./><  0.01 

Based  on  the  consistently  strong,  vmiformly  positive,  and  statistically  significant 
relationships  observed  between  item  scores  on  the  high  quality  of  experience  scale  and 
the  same  item’s  scores  on  the  low  quality  of  experience  scale,  the  findings  suggest  that 
the  two  measurement  scales  do  not  produce  independent  results.  Upon  closer 
examination  of  the  correlation  matrix  in  Appendix  B,  it  becomes  clear  virtually  all  item 
scores  significantly  and  positively  correlate  with  other  survey  item  scores  (p  <  0.05  or 
less),  regardless  of  whether  the  item  pertained  to  high  or  low  quality  of  experience.  Once 
again,  the  reader  is  cautioned  to  bear  in  mind  the  possibility  of  acquiescence;  in  which 
case,  the  systematic  errors  in  item  responses  would  correlate  with  each  other,  also 
accounting  for  high  inter-item  correlation. 

The  preponderance  of  positive  and  statistically  significant  inter-item  correlation 
eoefficients  (assuming  non-acquiescence),  the  lack  of  any  significant  factor  breakout 
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between  the  individual  survey  items  (eissuming  none  of  the  potential  factor-related 
problems  discussed  earlier),  and  the  high  estimates  of  internal  consistency  provided  by 
each  scale’s  respective  Cronbach’s  alpha,  all  led  to  the  conclusion  that  the  scales 
developed  and  used  in  this  study  were  most  likely  unidimensional  and  unidirectional,  and 
that  the  test  items  were  not  independent.  Restated  within  the  context  of  this  study,  there 
is  evidence  of  a  halo-like  effect,  suggesting  subjects’  individual  item  responses  either 
influenced,  or  were  strongly  influenced  by,  their  responses  to  most  (if  not  all)  of  the  other 
survey  items. 

One  could  argue  these  results  were  perhaps  attributable  to  recency  or  ordering 
effects  because  the  low  quality  of  experience  scale  was  administered  on  the  back  side  of 
the  same  form  as  the  high  quality  of  experience  scale,  and  (presumably)  immediately 
following  the  completion  of  the  high  quality  of  experience  scale.  However,  the  large 
number  of  positive  and  statistically  significant  inter-item  correlation  scores  between 
items  on  the  same  scales  reduces  the  likelihood  that  such  effects  appreciably  impacted  the 
results.  Thus,  the  assumption  of  independent  measures  does  not  appear  to  be  justified;  in 
other  words,  even  within  the  same  scales,  subjects  did  not  seem  to  differentiate  their 
considerations  for  single  item  responses  independently  of  their  other  item  responses. 
Assessing  the  Direction  of  Differences  Between  High  and  Low  Quality  Item  Responses 

Assuming  subjects  were  not  very  discerning  when  considering  their  responses  to 
high  and  low  quality  of  experience  survey  items  (but  did  not  acquiesce  to  the 
measurement  task  in  general,  as  was  discussed  earlier),  the  question  still  remained — 
which  items,  if  any,  were  more  important  to  perceptions  of  high  or  low  quality  of 
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experience?  Initially,  a  paired-samples  t-test  was  employed  to  answer  this  question. 
Based  on  those  initial  results,  all  but  one  of  the  mean  responses  between  matched  high 
and  low  quality  survey  items  differed  significantly  at/?  <  ,05.  However,  due  to  the 
evidence  suggesting  non-independent  test  items,  and  the  desire  to  remain  as  conservative 
as  possible  before  reporting  significant  findings  based  on  such  exploratory  research 
efforts,  t-test  results  were  corrected  for  multiple  simultaneous  comparisons  using  the 
Bonferroni  procedure,  thus  maintaining  an  experiment-vrise  alpha  of  less  than  0.05. 

Mean  squared  error  was  determined  for  each  high-low  quality  pair  of  survey  items  using 
a  simple  one-way  ANOVA  and  then  incorporated  into  the  Bonferroni  correction  formula: 

(Yi  -  Yj  )  +/-ta/(2c)  S  /(l/tli+l/llj) 

Significant  Bonferroni-corrected  confidence  intervals  for  the  mean  difference  between 
high  and  low  quality  of  experience  item  pairs  are  presented  in  Table  5. 


Table  5.  Bonferroni-corrected  Confidence  Intervals  for  Item  Pair  Mean  Differences 


Item  Pair 

Lower  Bound 

Unner  Bound 

HQ  2  - LQ  2 

0.111424 

1.02036 

HQ  4  - LQ  4 

0.136998 

1.13018 

HQ  6  - LQ  6 

0.02958 

0.87268 

HQ  8  - LQ  8 

0.073078 

0.96509 

HQ  9  - LQ  9 

0.104871 

1.05543 

HQ  11 -LQ  11 

0.054334 

0.9983 

HQ  12  -  LQ  12 

0.133229 

1.05474 

HQ  14-LQ14 

0.053339 

0.98426 

HQ  15 -LQ  15 

0.03761 

1.06163 

HQ  17 -LQ  17 

0.060861 

1.24741 

HQ  19 -LQ  19 

-2.09733 

-0.6997 
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Using  the  confidence  interval  surrounding  item-pair  mean  differences  as  a 
surrogate  measure  of  the  difference  in  magnitude  of  an  item’s  “importance”  to  overall 
perceptions  of  high  versus  low  quality  of  experience,  the  observed  pattern  of  results 
supports  the  following  conclusions: 

A.  Within  the  sampled  population. . . 

1 )  relevance  of  information  to  the  task 

2)  search  engine  options  (ex.  multiple  criteria) 

3)  amount  of  information  available 

4)  organization  of  information  at  web  sites 

5)  ease  of  use  of  browser 

6)  reliability  of  connection  (crashes,  disconnects) 

7)  clarity  of  directions  for  navigation  at  web  sites 

8)  up-to-date  links 

9)  availability  of  information  at  remote  servers 

10)  security/privacy 

...are  all  considered  relatively  more  important  to  perceptions  of  high  quality  of 

experience  than  to  perceptions  of  low  quality  of  experience. 

B.  Advertising  presence  appears  to  be  the  only  item  which  has  greater  influence 
on  overall  perceptions  of  low  quality  versus  high  quality  of  experience. 

Within  this  same  conceptual  framework,  those  item-pairs  in  which  the  confidence  interval 

of  the  mean  difference  encompassed  zero  should  not  be  considered  unimportant  to 

perceptions  of  high  or  low  quality  of  experience,  only  that  their  relative  importance  for 

influencing  perceptions  of  either  high  or  low  quality  of  experience  did  not  seem  to  differ 

in  a  statistically  significant  fashion.  Therefore,  if  the  overriding  goal  of  the  current 

investigation  is  to  construct  a  questionnaire  which  best  differentiates  the  most  salient 

characteristics  or  circumstances  which  influence  perceptions  of  high  quality  versus  low 
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quality  of  experience,  the  obtained  results  suggest  that  the  eleven  items  listed  above  have 
the  greatest  potential  for  supporting  such  quality  assessments. 

Exploring  the  “Critical”  Quality  of  Experience  Items 

In  order  to  produce  meaningful  data  from  a  survey  which  will  hopefully  measure 
the  relative  efficacy  of  IT-related  behavioral  determinants  over  time,  that  survey  should 
encompass  those  relevant  factors  which  exert  varying  degrees  of  behavioral  influence 
over  the  course  of  experience.  However,  the  factors  themselves  should  not  change  over 
time.  If  the  factors  did  change  over  time,  a  survey  based  upon  those  factors  would  not  be 
an  efficient  measurement  tool  because  some  items  would  gain  or  lose  absolute  relevance 
with  variations  in  user  experience  (little  value  would  be  gained  by  including  survey  items 
which  would  not  apply  to  the  population  of  interest  after  the  first  few  uses).  Thus,  in  the 
case  of  quality  of  experience,  those  items  which  appeared  to  be  relevant  concerns  for  both 
experienced  and  inexperienced  users,  an  J  the  relative  effects  of  which  seemed  capable  of 
differentiating  between  perceptions  of  high  quality  of  experience  and  low  quality  of 
experience,  were  considered  for  further  study. 

Given  the  discussion  in  the  preceding  section  of  this  analysis,  the  1 1  items  which 
exhibited  significant  mean  differences  fulfilled  each  of  the  criteria  noted  above:  all  items 
were  apparently  perceived  as  important  determinants  to  perceptions  of  quality  of 
experience,  and  the  relative  differences  between  the  item  pairs  seemed  to  differentiate 
item  scores  which  pertained  to  either  high  quality  of  experience  or  low  quality  of 
experience.  Although  item  19  has  already  been  shown  to  adversely  affect  the  internal 
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consistency  of  the  quality  scales,  it  was  nevertheless  retained  again  for  further  study 
simply  out  of  theoretical  curiosity.  ’ 

To  test  the  stability  of  these  items’  relative  influences  across  users  of  varying 
experience  levels,  subjects  were  divided  into  upper  and  lower  thirds  based  on  their 
reported  experience  levels.  Traditionally,  a  median  split  might  be  employed  for  such  a 
dichotomous  group-v^se  comparison  between  experienced  and  inexperienced  users;  again 
however,  using  the  upper  and  lower  extremes  for  inter-group  comparisons  seemed  a  more 
conservative  approach.  Eleven  users  with  0-6  months  of  Internet  experience  were 
combined  with  the  13  users  at  the  6  months  - 1  year  experience  level  to  form  the 
inexperienced  user  group;  all  22  subjects  reporting  3  or  more  years  of  Internet  experience 
were  used  for  the  high  experience  group. 

Results  of  one-way  ANOVA  for  both  high  quality  of  experience  items  and  low 
quality  of  experience  item  responses  revealed  no  significant  main  effects  for  user 
experience  at  either  unadjusted  or  Bonferroni-corrected  alpha  levels  in  all  but  one  of  the 
sample  comparisons  (p  <  0.1).  Fortunately,  this  sample  item  was  item  19,  “Advertising 
Presence,”  an  item  shown  time  and  again  to  have  effects  ranging  from  inconsequential  to 
detrimental  on  the  measurement  data  produced  by  this  particular  survey.  Means 
comparison  revealed  that  between  the  sampling  of  experienced  and  inexperienced  users, 
mean  item  responses  for  “Advertising  Presence”  were  significantly  higher  (p  <  0.001)  for 
inexperienced  users  (3.25)  when  they  pertained  to  high  quality  of  Internet  experience  than 
for  experienced  users  (1 .64).  It  should  be  noted,  however,  that  while  these  differences 
were  statistically  significant,  the  anchors  to  which  these  scores  refer  indicate  that  Internet 
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advertising  is  perceived  as  roughly  “Somewhat  Unimportant”  for  determining  high 
quality  of  experience  for  inexperienced  Internet  users,  and  somewhere  between  “Very 
Unimportant”  to  “Not  at  all  Important”  for  experienced  Internet  users. 

Reconciling  Item  19.  “Advertising  Presence” 

The  curious  case  of  item  19  actually  becomes  clear  in  light  of  the  results  reported 
above.  Based  on  the  obtained  survey  responses,  it  is  reasonable  to  assume  advertising 
presence  may  well  have  some  isolated  effects  on  perceptions  of  high  or  low  quality  of 
experience  when  using  the  Internet.  This  conclusion  was  echoed  in  the  notes  and 
comments  scribbled  on  several  of  the  returned  questionnaires  indicating  that  Internet 
advertising  is  generally  a  nuisance.  Statistically  speaking,  these  perceptions  may  even 
vary  in  a  significant  manner  between  users  of  various  experience  levels. 

However,  when  assessing  overall  perceptions  of  quality  of  experience,  as  captured 
by  the  other  apparently  relevant  survey  items,  Internet  advertising  appears  to  have  little 
consistent  or  systematic  effect.  In  fact,  the  findings  reported  above  were  not  at  all 
surprising  given  the  large  standard  deviation  of  the  mean  difference  for  the  item  19  high- 
low  quality  pair  (2.31)  relative  to  every  other  paired  difference  (range:  1 . 1 9  to  1 .48). 
Quite  simply,  the  effects  of  Internet  advertising,  at  least  within  the  context  of  this 
investigation,  appeared  to  have  been  too  variable  and  inconsistent  to  have  any  practical  or 
theoretical  relevance  to  subsequent  investigations  of  the  quality  of  experience  construct. 
Synthesis 

To  summarize  the  implications  of  the  findings  reported  in  the  more  recent 
sections  of  this  chapter,  the  obtained  results  suggest  that  1 1  of  the  19  survey  items  are 
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capable  of  differentiating  overall  patterns  of  scores  relating  to  high  versus  low  quality  of 
experience.  However,  10  of  the  1 1  items  were  not  foimd  to  differ  significantly  in  the 
subjects’  appraisal  of  each  items  relative  importance  for  determining  high  or  low  quality 
of  experience.  This  further  suggests  this  core  set  of  10  items  could  indeed  represent  a 
stable  set  of  (quality-related)  criteria,  each  statistically  equal  in  importance  to  both 
experienced  and  inexperienced  IT  users,  and  that  the  values  returned  by  taking 
measurements  of  these  criteria  can  indicate  the  degree  to  which  the  IT  usage  experiences 
have  been  of  high  or  low  quality. 

To  assess  the  degree  to  which  these  conclusions  were  warranted,  a  follow-up 
analysis  of  internal  consistency  for  both  high  and  low  quality  of  experience  was 
conducted  using  only  the  10  critical  survey  items  noted  above.  Cronbach’s  alpha  for  the 
revised  high  quality  scale  was  0.85,  only  a  marginal  drop  in  estimated  internal 
consistency  from  the  original  alpha  of  0.87  reported  for  the  full  19-item  scale.  The 
internal  consistency  estimate  for  the  low  quality  of  experience  scale  was  0.89,  again  only 
a  marginal  drop  in  internal  consistency  from  the  original  full-scale  alpha  statistic  of  0.92. 

Thus,  the  evidence  does  indeed  suggest  that  a  measurement  scale  based  on  the 
following  10  survey  items  could  be  a  reasonably  consistent  instrument  for  assessing 
perceptions  of  quality  of  experience: 

1 )  relevance  of  information  to  the  task 

2)  search  engine  options  (ex.  multiple  criteria) 

3)  amount  of  information  available 

4)  organization  of  information  at  web  sites 

5)  ease  of  use  of  browser 

6)  reliability  of  connection  (crashes,  disconnects) 

7)  clarity  of  directions  for  navigation  at  web  sites 
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8)  up-to-date  links 

9)  availability  of  information  at  remote  servers 

10)  security/privacy 


Possible  anchors  for  subsequent  use  of  the  modified  quality  of  experience  scale  are 
addressed  in  the  next  chapter,  as  are  the  issues  of  whether  or  not  the  “right”  questions 
appear  on  the  scale,  or  if  the  questions  should  even  be  asked  at  all. 
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V.  Discussion 


Relevance  of  the  Current  Investi2ation 

A  recurring  theme  throughout  the  course  of  this  study  was  the  running  assumption 
that  whatever  traits,  factors,  or  characteristics  the  quality  of  experience  survey  eventually 
captured  actually  have  at  least  some  bearing  on  IT  usage  behaviors.  Based  on  the 
obtained  results,  there  is  evidence  to  suggest  that  the  10  critical  quality  of  experience 
survey  items  do  encompass  a  set  of  stable,  meaningful,  and  consistent  traits  or 
characteristics,  commonly  understood  to  be  related  to  quality  of  experience,  and  shared 
across  user  groups  of  varying  Internet  experience.  Thus,  the  first  step  towards 
imderstanding  IT  use  within  the  conceptual  framework  of  the  quality  of  experience 
construct  has  been  taken:  establishing  the  key  traits  and  characteristics  Internet  users 
seem  to  associate  with  experiential  quality.  Determining  the  extent  to  which  those  traits 
or  characteristics  actually  influence  IT-related  behaviors  requires  additional  investigation; 
suggestions  for  such  efforts  are  discussed  in  the  following  sections. 

However,  even  without  relating  the  results  of  this  study  directly  back  to  IT  use  at 
this  time,  the  quality  of  experience  construct,  as  it  has  been  defined  within  the  context  of 
this  study,  still  seems  to  convey  a  certain  amount  of  pertinent  information  concerning  IT 
implementation  (or  at  least  information  concerning  Internet-related  implementations  of 
IT).  From  a  descriptive  standpoint,  obtained  results  suggest  that  if  the  experiences  of  the 
sampled  Internet  users  are  to  be  labeled  or  characterized  as  high  or  low  quality,  the  events 
or  circumstances  which  transpired  over  the  course  of  those  users’  experiences  with  the 
Internet  must  have  pertained  to  issues  associated  with: 
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1)  relevance  of  information  to  the  task 

2)  search  engine  options  (ex.  multiple  criteria) 

3)  amount  of  information  available 

4)  organi2ation  of  information  at  v^eb  sites 

5)  ease  of  use  of  browser 

6)  reliability  of  connection  (crashes,  disconnects) 

7)  clarity  of  directions  for  navigation  at  web  sites 

8)  up-to-date  links 

9)  availability  of  information  at  remote  servers 

10)  security/privacy 

Therefore,  from  a  prescriptive  standpoint  for  IT  implementation,  results  of  the 
current  study  suggest  that  the  hallmarks  of  providing  high  quality  Internet  experiences 
include  offering  facilities  such  as  flexible  search  engines  with  several  options  for 
conducting  searches,  browsers  which  are  easy  to  use,  avoiding  referrals  to  missing  or 
outdated  information,  ensuring  a  well-organized  presentation  of  relevant  information,  and 
other  factors  explicit  in  the  items  listed  above.  In  addition,  the  sampled  Internet  users 
appeared  to  be  most  sensitive  to  variations  in  these  particular  factors.  Thus,  changing 
perceptions  of  quality  in  a  particular  situation,  or  perhaps  making  comparative,  quality- 
related  judgments  between  two  potential  Internet  interfaces,  would  most  likely  be 
accomplished  by  affecting  changes  in,  or  taking  measurements  of,  these  10  particular 
aspects  of  the  Internet  experience. 

Reflections  on  Obtained  Response  Data 

A  modicum  of  faith  in  the  respondents’  intentions  was  necessary  to  make  any 
meaningful  interpretations  of  the  obtained  results.  For  example,  the  research  assumes 
that  subjects  were  thoughtful  and  discriminating  in  their  item  responses — ^responses 
which  were  based  on  their  actual  feelings  about  each  item’s  relative  importance  to  overall 
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perceptions  of  quality  of  experience.  Under  such  an  assumption,  the  apparent 
unidimensionality  of  the  surveys  suggest  subjects  were  not  sensitive  to  the  apparent 
discord  amongst  the  items’  content  and  did  not  systematically  group  them  into  separate 
“clusters”  of  issues  which  were  more  or  less  important  than  others  (for  example, 
groupings  of  hardware  versus  software  versus  interface-related  items).  Thus,  for  the 
purposes  of  construct  and  instrument  development,  this  lack  of  responsiveness  or  inter¬ 
item  discrimination  was  not  interpreted  to  mean  subjects  were  not  careful  about  choosing 
the  responses  they  did,  rather,  that  the  survey  items  themselves  reflected  quality-related 
issues  which  were  not  considered  independently  of  others. 

Yet,  it  may  be  naive  not  to  suspect  such  confounding  mechanisms  at  work. 
Indeed,  there  is  the  very  real  possibility  subjects  were  not  careful  and  discriminating  in 
their  item  responses  due  to  the  lack  of  concrete  response  stimuli.  As  discussed  earlier  in 
Chapter  IV,  this  could  also  account  for  the  indications  of  unidimensionality.  Such 
concerns  should,  therefore,  not  be  overlooked  during  follow-on  research.  However, 
assuming  a  shorter  survey  would  encourage  more  people  to  participate,  and  perhaps  a 
more  cogent,  task-specific  stimulus  would  encourage  them  to  thoughtfully  answer  the 
survey,  use  of  the  revised  10-item  quality  of  experience  scale  might  allow  future 
investigators  to  discount  some  of  the  potential  effects  of  these  issues  on  their  results. 
Quality  of  Experience — Finding  the  Right  Dependent  Variable 

As  mentioned  during  the  analysis  section  of  this  report,  the  ultimate  goal  of  this 
exploratory  study  was  to  identify  a  factor  or  construct  capable  of  accounting  for  changes 
in  IT  user  behaviors  over  time.  Accomplishing  this  objective  means  ultimately  relating 
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quality  of  experience  back  to  IT  user  behaviors.  Future  study  within  this  research  stream 
must  therefore  focus  on  pairing  or  correlating  various  scores  obtained  on  the  quality  of 
experience  survey  items  with  one  or  more  dependent  measures  of  IT  acceptance  or  use. 

Such  comparative  statistics  would  allow  for  a  number  of  refinements  in  the 
current  state  of  quality  of  experience  instrument.  First,  it  would  allow  for  reassessment  of 
the  umdimensionality  of  the  quality  of  experience  scale,  as  is  currently  suggested  by  the 
obtained  results;  that  is,  by  relating  the  individual  item  scores  on  the  survey  to  other  IT- 
related  measurements,  the  extent  to  which  all  the  quality  of  experience  items  are 
measuring  the  same  cognitive  phenomenon  can  be  empirically  tested. 

Second,  it  would  help  indicate  what  various  scores  on  the  quality  of  experience 
questionnaire  actually  mean,  or  should  mean,  in  terms  of  the  quality  of  experience 
construct.  Are  high  scores  more  predictive  of  high  rates  of  IT  use  than  low  scores  of  low 
rates  of  use?  How  do  we  compute  such  scores  to  begin  with;  can  we  simply  add  the  item 
scores  together  to  produce  a  “quality  index”  of  some  kind?  These  are  all  questions  about 
the  quality  of  experience  scale  which  must  be  answered  before  definitively  establishing 
the  quality  of  experience  construct  as  a  valid  concern  to  the  subject  of  IT  use. 
Reconsidering  the  Notion  of  Two  Types  of  Quality  of  Experience 

Also  mentioned  in  previous  sections  of  this  report,  the  notion  of  “quality”  has  the 
potential  to  capture  or  embody  many  different  meanings.  Logically  then,  the  notion  of 
“high  quality”  most  likely  has  the  potential  to  mean  many  different  things  beyond  simply 
“the  conditions  opposite  of  low  quality.”  Until  such  time  as  the  evidence  suggested 
otherwise,  this  was  assumed  to  be  so.  However,  in  light  of  the  findings  obtained  during 
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the  course  of  the  current  investigation,  the  benefits  derived  from  continued  adherence  to 
such  an  assumption  must  now  be  questioned. 

Given  the  current  pattern  of  results,  there  is  reason  to  believe  both  quality  of 
experience  scales  may  exhibit  some  degree  of  unidimensionality,  unidirectionality,  and 
internal  consistency — ^at  least  within  the  context  of  this  study,  people  who  tended  to 
answer  high  on  the  relative  importance  of  one  particular  item  to  perceptions  of  high 
quality  also  tended  to  answer  high  on  the  relative  importance  of  the  same  item  to 
perceptions  of  low  quality,  as  well  as  answer  consistently  high  on  most  other  items 
(again,  however,  these  could  also  be  tell-tale  signs  of  acquiescence — prefer  to  the  next 
section  for  additional  discussion).  In  addition,  the  variability  between  scores  on  the 
critical  10  survey  items  seemed  to  be  able  to  significantly  differentiate  between  response 
sets  corresponding  to  overall  perceptions  of  high  quality  versus  those  pertaining  to 
overall  perceptions  of  low  quality. 

Consequently,  it  is  possible  that  future  use  of  the  quality  of  experience  scale  may 
not  have  to  be  saddled  by  a  two-factor  approach  to  the  quality  of  experience  construct. 
Based  on  the  responses  obtained  from  this  sample  of  Internet  users,  statistically 
significant  differences  between  item  scores  for  the  critical  10  quality  of  experience  items 
were  attributable  only  to  whether  or  not  the  items  related  to  high  or  low  quality  of 
experience;  other  potential  sources  of  variability,  such  as  the  users’  level  of  experience, 
did  not  seem  to  moderate  the  relative  importance  of  these  items  to  overall  perceptions  of 
quality  of  experience.  Therefore,  it  is  reasonable  to  expect,  given  the  appropriate  scale 
anchors,  that  only  one  set  of  survey  items  would  otherwise  be  necessary  to  capture  an 
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Internet  user’s  perceptions  of  overall  quality  of  experience — ^the  scores  obtained  on  those 
items  would  then  be  the  keys  to  indicating  whether  those  perceptions  were  of  high  or  low 
quality.  For  example,  each  of  the  10  critical  items  could  be  measured  against  a  semantic 
differential  scale  (anchored  at  the  extremes  by  high  quality  and  low  quality)  such  that 
respondents  were  forced  to  simultaneously  consider  an  item’s  relative  influence  on  their 
perceptions  quality  of  experience  at  the  time. 

Defining  the  Context  of  the  New  Quality  of  Experience  Scale — ^The  Crucial  Experiment 

Throughout  the  course  of  this  and  the  preceding  chapters,  it  has  been  suggested 
that  the  quality  of  experience  scales  may  be  unidimensional.  Given  the  validity  of  a 
number  of  assumptions,  the  evidence  certainly  lends  credence  to  such  inferences. 
However,  it  has  not  been  the  intent  to  trivialize  the  possibility  that  these  findings  were 
simply  artifacts  of  research  design  limitations;  nor  would  a  responsible  researcher  wish  to 
ignore  the  possibility  of  discovering  such  disconfirming  evidence.  Therefore,  a  crucial 
experiment  is  strongly  encouraged  to  challenge  the  degree  to  which  the  findings  obtained 
in  this  study  actually  reflect  a  stable  and  legitimate  psychological  phenomenon. 

The  first  issue  which  must  be  addressed  is  the  possibility  of  acquiescence  due  to  a 
lack  of  concrete  stimuli.  Recall  that  for  the  exploratory  purposes  of  this  investigation, 
only  quality-related  perceptions  subject  to  the  net  effects  of  all  the  users’  Internet 
experiences  were  accommodated;  no  contextual  assumptions  were  made  or  implied. 
However,  a  general  understanding  of  quality  of  experience  is  of  questionable  value  if  it 
cannot  be  applied  within  the  context  of  a  specific  task  environment  of  relevance  to  the  IT 
planner/manager — an  assessment  of  a  particular  Web  site  or  interface  design,  for 
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example.  As  such,  it  is  incumbent  upon  those  who  would  employ  the  quality  of 
experience  scale  for  practical  applications  to  realize  that  the  various  facets  of  quality 
suggested  in  this  study  may  not  be  as  salient  or  relevant  to  individual  Internet  experiences 
as  they  appeared  to  be  for  overall  (perhaps  even  “lifetime”)  appraisals  of  Internet 
experience.  Thus,  one  aspect  of  the  crucial  experiment  should  include  some  contextual 
considerations  of  a  specific  Internet  experience,  either  during  instances  of  actual  use,  or  a 
contrived  setting  in  which  the  experimenter  has  control  over  various  quality-related 
aspects  of  the  experience  itself 

The  next  issue  which  must  be  challenged  is  the  notion  of  unidimensionality.  As 
suggested  in  Chapter  IV,  it  is  possible  that  the  factor  analysis  did  not  extract  smaller, 
discrete  facets  within  the  quality  of  experience  scales  because  any  such  facets  were 
captured  within  the  context  of  only  a  few  survey  items.  Consequently,  a  follow-on 
investigation  could  test  this  implied  unidimensionality  by  producing  a  measurement 
instrument  comprised  of  separate,  unidimensional  sub-scales,  each  of  which  were 
consistent  with  the  items  this  study  suggests  are  associated  with  quality  of  experience. 

The  following  is  an  illustrative  example  of  such  sub-scale  development  activities. 

Consider  these  items  fi-om  the  quality  of  experience  survey:  “up-to-date  links,” 
and  “clarity  of  directions  for  navigation  at  web  sites.”  Both  seem  to  suggest  something 
about  navigation,  movement  from  or  through  one  source  of  information  to  another. 
Therefore,  “navigation”  could  be  a  potential  facet  of  the  quality  of  Internet  experiences. 

A  navigation  sub-scale  could  then  be  developed  using  these  two  items  as  conceptual 
guides  for  additional  item  selection.  For  example,  “niunber  of  navigational  steps  required 
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to  reach  the  desired  information”  would  seem  to  be  a  reasonable  item  to  include  on  this 
sub-scale,  as  would  “ease  with  which  desired  information  was  foxmd.”  Perhaps 
“orgamzation  of  information  at  web  sites”  also  pertains  to  navigational  factors.  Follow- 
on  research  should  attempt  to  develop  sub-scales  which  are  themselves  unidimensional 
(measuring  the  same  thing),  and  exhibit  very  high  inter-item  correlations  (subjects  tend  to 
answer  similarly  and  consistently  within  the  sub-scale). 

By  combining  these  new  sub-scales  with  a  concrete  stimulus,  the  degree  to  which 
conclusions  drawn  in  this  study  were  warranted  could  be  assessed.  For  example,  the 
navigational  sub-scale  might  be  constructed  in  a  manner  similar  to  the  following: 

Please  rate  the  navigational  features  of  the  Internet  site  you  just  visited  along  the 

following  dimensions: 

Up-to-date  links 

LowQuality _ : _ : _ : _ ; _ : _ : _ HighQuality 

1  2  3  4  5  6  7 

Clarity  of  directions  for  navigation  at  web  site 

LowQuality _ : _ : _ : _ : _ : _ : _ HighQuality 

1  2  3  4  5  6  7 

Number  of  navigational  steps  required  to  reach  the  desired  information 

LowQuality _ : _ : _ : _ : _ : _ ; _ HighQuality 

1  2  3  4  5  6  7 

...and  so  forth. 

If  the  results  of  such  an  investigation  revealed  fluctuations  in  item-total 
correlations  (item-total  correlations  in  this  study  were  predominantly  unidirectional). 
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strong  factor  loadings  along  the  specified  sub-scale  boundaries,  and  high  inter-item 
correlations  within  each  sub-scale,  these  findings  would  essentially  provide  a  counter¬ 
example  to  the  notion  that  quality  of  experience,  as  defined  within  the  context  of  this 
exploratory  study,  is  unidimensional.  However,  should  the  results  of  such  a  study 
indicate  each  of  the  sub-scales  were  themselves  highly  inter-correlated,  they  would 
provide  very  compelling  evidence  for  the  unidimensionality  of  quality  of  experience  as 
proposed  here.  Findings  of  this  nature  would  suggest,  within  yet  another  investigative 
context,  that  although  Internet  users  may  associate  many  different  circumstances,  traits, 
or  conditions  with  their  perceptions  of  quality,  they  do  not  differentiate  amongst  those 
items  in  terms  of  relative  influence  on  their  perceptions. 

The  remainder  of  this  chapter  examines  additional  limitations  of  the  current 
investigation.  Several  disparate  theoretical  backdrops,  both  from  IT-based  and  behavioral 
and  cognitive  science-based  perspectives,  will  then  be  advanced  for  the  purposes  of 
interpreting  the  obtained  results.  However,  at  no  time  should  the  conclusions  drawn 
during  the  course  of  this  discussion  be  considered  anything  more  than  speculative  and 
preliminary.  Until  such  time  as  a  crucial  experiment  is  conducted  which  confirms  or 
disconfirms  some  of  the  more  fundamental  aspects  of  quality  of  experience,  the 
subsequent  analysis  which  appears  below  should  always  be  considered  in  light  of  the 
potential  limitations  discussed  in  the  sections  above. 

Sample  Size  Limitations 

Although  the  desired  sample  size  was  not  obtained  during  the  course  of  this  study, 
limited  and  conservative  interpretations  within  the  sample  did  at  least  hint  at  the 
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possibility  of  a  unidimensional  construct.  Certainly,  the  small  n  may  have  been 
especially  problematic  given  the  lack  of  concrete  stimulus  (i.e.  the  current  investigation 
used  subjects’  lifetime  of  Internet  experiences  versus  a  single,  specific,  and  perhaps  even 
contrived  or  mampulated — ^Internet  session).  Similarly,  the  additional  statistical  power 
gained  by  a  larger  n,  the  use  of  multiple  items  for  each  conceivable  facet  of  quality,  as 
well  as  the  added  external  validity  of  an  even  larger  and  more  heterogeneous  subject  pool, 
would  enhance  the  standing  of  quality  of  experience  as  a  construct  of  relevance  to  the 
subject  of  IT  use. 

Despite  concerns  over  sample  size,  there  are  reasons  to  believe  the  obtained  n  was 
not  a  serious  limitation  for  this  study.  A  brief  survey  of  the  relevant  IT-related  research 
reveals  many  studies  have  been  conducted  using  a  relatively  small  sample  size:  Davis  et 
al.’s  (1989)  findings  were  based  on  the  responses  of  only  107  IT  users,  Adams  et  al. 
(1992),  only  118.  Even  Davis’  (1989)  initial  foray  into  the  usefulness  and  ease  of  use 
constructs,  the  very  foundations  of  the  highly  influential  TAM,  was  based  upon  the 
responses  of  only  152  total  users  over  the  course  of  two  separate  studies.  Therefore,  the 
conclusions  based  upon  results  obtained  in  this  study— despite  the  small  sample  size— 
appear  reasonable,  at  least  within  the  investigative  context  defined  by  other  related 
studies.  Furthermore,  the  conservative  statistical  correction  factors  used  to  maintain  an 
acceptably  low  experiment-wise  alpha  offer  some  assurance  that  the  statistically 
significant  effects  observed  in  this  study  may  indeed  reflect  actual  and  appreciable 
differences  within  the  sampled  population. 
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Cross-sectional  versus  Longitudinal  Samples 


Because  this  investigation  was  not  designed  for  repeated  measures,  conclusions 
pertaining  to  the  stability  of  the  critical  quality  of  experience  items  over  time  had  to  be 
inferred  from  the  item  responses  of  users  at  different  experience  levels.  That  no 
significant  differences  were  observed  between  relative  item  importance  scores  as  a 
function  of  experience  provided  initial  indications  that  the  components  of  whatever 
imderlying  construct  was  being  measiued  via  the  quality  of  experience  survey  remained 
stable  over  time.  Coupled  with  the  conservative  statistical  approach  used  throughout  the 
analysis,  the  evidence  suggests  (acknowledging  the  potential  for  acquiescence)  that  the  10 
critical  items  do  capture  relevant  quality-related  concerns  which  are  weighted  just  as 
importantly  to  inexperienced  users  as  they  are  to  experienced  users.  However,  the  degree 
to  which  this  conclusion  applies  within  groups  has  yet  to  be  addressed. 

It  has  already  been  suggested  that  scores  obtained  using  the  quality  of  experience 
scale  should  be  paired  with  measurements  of  actual  use  to  test  the  efficacy  of  the  quality 
of  experience  construct  for  explaining  changes  in  Internet  usage  behavior.  Future 
investigators  should  also  consider  taking  repeated  measurements  within  the  same 
population  sample,  thus  allowing  for  estimates  of  how  well  perceptions  of  quality  of 
experience  predict  subsequent  usage  behaviors  as  experience  levels  themselves  change. 
Increasing  the  Clarity  of  the  Critical  Survey  Items 

On  a  more  subjective  aside,  further  attention  regarding  quality  of  experience  and 
the  efficacy  of  the  critical  10  quality  of  experience  items  should  include  consideration  of 
each  item’s  clarity  and  interpretability.  In  the  original  survey  mailing,  it  was  deemed 
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appropriate  to  make  each  item  as  short  as  possible,  both  to  fit  all  items  on  one  side  of  a 
sheet  of  paper,  and  to  keep  the  entire  survey  short  enough  to  encourage  participation.  It 
was  possible  some  tradeoff  between  clarity  and  brevity  was  made  in  the  process. 
However,  the  obtained  evidence  now  suggests  that  a  revised  survey  need  only  include  10 
of  the  original  19  items,  and  that  only  one  assessment  of  overall  quality  (versus  separate 
assessments  of  high  and  low  quality)  need  be  addressed  within  the  survey  itself. 

Clearly,  this  allows  for  the  possibility  of  more  fully  expanding  the  content  of  each 
of  the  critical  quality  of  experience  items.  But  is  such  attention  warranted  given  the 
results  obtained  in  this  study?  For  example,  “Availability  of  information  at  remote 
servers”  was  intended  to  refer  to  those  situations  in  which  the  user  knows  the  desired 
information  is  available  (or  is  told  as  such),  but  is  unable  to  retrieve  the  information  from 
the  appropriate  source.  Estimates  of  internal  consistency  within  the  revised  10-item  scale 
suggest  even  when  users  with  0-6  months  of  Internet  experience  were  included  in  the 
subject  pool,  this  item  generated  consistent  responses  (consistent  interpretation  was 
inferred  from  these  results).  Therefore,  it  is  reasonable  to  assume  that  the  clarity  of  the 
survey  items  did  not  have  a  confounding  effect  during  this  investigation,  although  such 
an  assumption  must  be  reexamined  within  the  context  of  an  additional  study.  Having 
subjects  rate  each  item’s  clarity  would  be  a  simple  means  of  addressing  this  issue. 
Prudence  also  suggests  particular  attention  be  paid  to  the  response  patterns  of  more 
inexperienced  users  as  the  potential  for  confusion  regarding  Internet-specific  items  is 
probably  greatest  for  this  portion  of  the  population. 
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Why  These  Critical  Items?  An  IT-Based  Perspective 


Before  further  considering  any  other  implications  of  the  reported  findings,  a 
fundamental  and  theoretically  relevant  question  must  be  posed:  why  were  these 
particular  items  found  to  be  significant  determinants  for  quality  of  experience?  Such  a 
question  can  only  be  answered  upon  careful  consideration  of  item  content. 

Turning  this  discussion  back  to  the  literature  review  in  Chapter  II,  recall  that 
TAM  postulates  perceived  ease  of  use  and  usefulness  as  significant  determinants  of  IT 
use.  It  is  conceivable  that  the  quality  of  experience  items  actually  capture  the  significant 
aspects  of  Internet-related  experiences  which  form  the  basis  of  these  core  IT-related 
beliefs.  For  example,  “Ease  of  use  of  browser”  was  one  of  the  critical  items  and  clearly 
reflects  the  ease  of  use  construct;  however,  it  is  not  difficult  to  see  how  “Clarity  of 
directions  for  navigation  at  web  sites”  could  also  relate  to  perceived  ease  of  use  (these 
items  might  even  cluster  together  as  a  facet  of  quality  when  the  instrument  is  used  to 
assess  specific  Internet  sessions  or  Web  sites). 

Similarly,  items  which  relate  to  perceived  usefulness  could  include  “Relevance  of 
information  to  the  task”;  “Search  engine  options”;  and  “Availability  of  information  at 
remote  sites.”  Upon  careful  inspection  of  the  critical  quality  of  experience  items,  there 
exists  the  possibility  that  all  of  these  items  provide  evidence  upon  which  Internet  users 
could  base  their  perceptions  of  ease  of  use  and  usefulness.  However,  such  post-hoc 
conclusions  may  be  criticized  for  being  too  convenient — ^that  the  similarities  between  the 
quality  of  experience  items  and  the  usefulness/ease  of  use  constructs  were  simply  “read 
into”  the  comparisons  after  the  fact.  Such  criticisms  may  even  be  warranted;  however. 
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other  research  findings  outside  the  realm  of  IT  use  and  TAM-related  investigations  also 
suggest  something  fundamental  about  the  nature  of  the  critical  quality  of  experience  items 
to  the  Internet  usage  experience. 

Levi  and  Conrad  (1996)  conducted  a  study  concerning  the  design  and  evaluation 
of  a  World  Wide  Web  prototype.  Based  on  their  results,  several  usability  principles  (or 
heuristics)  were  proposed,  the  following  of  which  are  relevant  to  the  current  discussion: 

1)  Build  flexible  and  efficient  systems:  accommodate  a  wide  range  of 
user  sophistication  and  diverse  user  goals.  Provide  instructions  where 
useful.  Lay  out  screens  so  that  frequently  accessed  information  is 
easily  found. 

2)  Provide  progressive  levels  of  detail:  organize  information 
hierarchically.  Encourage  the  user  to  delve  as  deeply  as  needed,  but  to 
stop  whenever  sufficient  information  has  been  received. 

3)  Give  navigational  feedback:  facilitate  jumping  between  related  topics. 

4)  Don ’t  lie  to  the  user:  eliminate  erroneous  or  misleading  links.  Do  not 
refer  to  missing  information.  (Levi  and  Conrad,  1996:58) 

Table  6  presents  a  subjective  comparison  between  these  heuristics  and  the  critical  quality 

of  experience  items.  For  the  purposes  of  this  comparison,  it  may  be  helpful  to  refer  back 

to  the  definitions  from  time  to  time. 


Table  6.  Potential  Matches  Between  Critical  Quality  of  Experience  Survey  Items  and 
World  Wide  Web  Prototype  Usability  Principles 


1  Usability  Principle 

Critical  Quality  of  Exnerience  Item 

Build  flexible  and  efficient  systems 

Search  engine  options 

Clarity  of  directions  for  navigation  at  web  sites 

Provide  progressive  levels  of  detail 

Organization  of  information  at  web  sites 

Amount  of  information  available 

Relevance  of  information  to  the  task 

Give  navigational  feedback 

Clarity  of  directions  for  navigation  at  web  sites 

Don’t  lie  to  the  user 

Availability  of  information  at  remote  servers 
Up-to-date  links 
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Again,  this  is  yet  another  “after-the-fact”  comparison;  however,  it  is  interesting 
nonetheless  to  note  the  parallels  between  Levi  and  Conrad’s  (1996)  usability  principles 
and  the  significant  quality  of  experience  survey  items  which  were  obtained  using  a 
completely  independent  and  unrelated  methodology.  Although  not  intentional,  the 
current  investigation  may  have  validated  some  of  Levi  and  Conrad’s  (1996)  prototype 
design  principles  within  the  context  of  another  user  population.  Conversely,  the  usability 
principles  may  indeed  reflect  some  fimdamental  properties,  inherently  desirable  for 
Internet  interfaces,  which  users  also  associate  with  perceptions  of  quality  of  experience. 

Further  evidence  of  the  consistency  and  relevance  of  the  critical  quality  of 
experience  items  to  the  subject  of  Internet  use  appears  in  Lightner  and  Bose’s  (1996) 
ergonomic  study  of  the  World  Wide  Web.  Using  an  item-generation  technique  similar  to 
the  CIT,  Lightner  and  Bose  (1996)  compiled  survey  responses  concerning  what  users 
liked  best  and  least  about  the  Internet,  and  where  they  experienced  the  most  difficulty  in 
using  the  Internet.  As  indicated  in  Table  7,  the  more  fi-equent  responses  to  their  survey 
were  remarkably  similar  to  the  critical  quality  of  experience  items  obtained  in  this  study. 


Table  7.  Comparison  of  Results:  Lightner  and  Bose  (1996)  and  Current  Investigation 


Liehtner  and  Bose  (1996) 
Ergonomic  Study  of  World  Wide  Web 

Turner  (1997) 

Quality  of  Internet  Experience 

Amount  of  information  available 

Amount  of  information  available 

Ease  of  use 

Ease  of  use  of  browser 

Searching  for  specific  information 

Relevance  of  information  to  the  task 

No  complete  category  search 

Search  engine  options 

Locating  and  navigating  sites 

Up-to-date  links 

Clarity  of  directions  for  navigation  at  web  sites 

Unavailability  of  sites 

Availability  of  information  at  remote  servers 

These  similarities  provide  further  reason  to  believe  that  users’  perceptions  of,  and 
feelings  towards,  Internet  use  are  predominantly  determined  by  a  relatively  smalls 
seemingly  stable,  and  apparently  consistent  set  of  traits,  conditions,  or  circumstances. 
Whether  the  culmination  of  these  traits,  conditions,  or  circumstances  should  survive  as 
the  notion  of  quality  of  experience,  as  has  been  suggested  by  the  results  of  this 
investigation,  remains  to  be  seen. 

Why  These  Critical  Items?  A  Behavioral  and  Cognitive  Science-Based  Perspective 

Until  now,  the  discussions  of  the  relevance  of  the  critical  quality  of  experience 
survey  items  and  the  validity  of  the  quality  of  experience  construct  have  had  a  distinctly 
IT-related  flavor.  However,  I  believe  the  study  of  attitudes  and  social  cognition  may  also 
provide  some  insight  as  to  nature  of  the  critical  quality  of  experience  survey  items  (and 
the  xmderlying  construct  presumably  tapped  by  these  items).  Specifically,  the  critical 
quality  of  experience  items  may  be  of  particular  import  to  perceptions  of  Internet  use  as  a 
result  of  the  cognitive  experience  of  regret. 

Through  a  series  of  telephone  interviews,  written  questionnaires,  and  face-to-face 
interviews,  Gilovich  and  Medvec  (1994)  found  that  the  experience  of  regret  seemed  to 
follow  a  systematic  time  course.  The  nature  of  this  temporal  effect  was  observed  in  that 
actions  which  have  led  to  misfortune  generally  cause  more  pain  in  the  short  term,  but  it 
was  inaction  on  the  part  of  the  subject  which  was  regretted  most  in  the  long  nm  (Gilovich 
and  Medvec,  1994:361).  Moreover,  Gilovich  and  Medvec  (1994)  propose  that  this  effect 
is  responsible  for  increasing  the  “cognitive  availability”  of  regrettable  inaction — ^that  we 
are  more  likely  or  more  frequently  able  to  remember  instances  or  circumstances  where 
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our  inaction  caused  us  to  experience  regret,  rather  than  our  actions  (Gilovich  and  Medvec, 
1994:363-364). 

These  cognitive  effects  of  regret  may  be  able  to  account  for  some  of  the  temporal 
variability  observed  within  the  various  TAM-related  studies  reported  in  Chapter  II. 

Recall  that  perceived  ease  of  use  was  defined  as  “the  degree  to  which  a  person  believes 
that  using  a  particular  system  would  be  free  of  effort”  (Davis,  1989:3^0).  This  implies 
that  although  a  system  might  not  be  easy  to  use,  you  could  still  accomplish  your  goals  by 
using  it.  Perceived  usefulness  was  defined  as  “the  degree  to  which  a  person  believes  that 
using  a  particular  system  would  enhance  his  or  her  job  performance”  (Davis,  1989:320). 
This  implies  that  regardless  of  how  easy  to  use  a  system  might  be,  if  you  cannot  do  what 
you  want  to  with  it,  you  will  not  consider  the  system  useful. 

Now  consider  the  notion  of  regrettable  actions  and  inaction.  Suppose  some 
particular  system  factors  reduced  ease  of  use.  The  resulting  confusion  or  incorrect 
actions  on  the  user’s  part  could  lead  to  frustration,  anger,  or  even  a  reduction  in  system 
use;  but  these  effects  would  probably  last  only  until  the  user’s  experience  level  was  such 
that  those  factors  no  longer  affected  performance.  However,  if  certain  factors  reduced  the 
usefulness  of  the  system  itself,  it  stands  to  reason  that  the  user  might  not  be  able  to 
accomplish  his  or  her  ultimate  goals  by  using  the  system.  Consequently,  those  factors 
relating  to  decreased  usefulness  would  have  contributed  to  the  user’s  inability  to  do 
something.  However,  that  inability  could  only  be  realized  or  recognized  after  the  user 
expended  some  effort  to  accomplish  his  or  goals  in  the  first  place. 
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According  to  Gilovich  and  Medvec’s  (1994)  theory  of  regret,  those  circumstances 
which  have  lead  to  the  inability  to  accomplish  something  (usefulness)  should  cause  more 
negative  affect  and  be  more  easily  and  frequently  recalled  than  those  which  have  caused 
actions  producing  imdesired  outcomes  (ease  of  use).  Therefore,  factors  which  reduce 
usefulness  should  lead  to  greater  regret  and  be  more  memorable  than  factors  which 
reduce  ease  of  use.  Clearly  then,  the  experience  of  regret  could  account  for  the  findings 
of  Davis  et  al.  (1989)  in  which  the  direct  effect  of  ease  of  use  on  behavioral  intention  was 
significant  for  inexperienced  users,  but  was  fully  mediated  by  usefulness  for  experienced 
users  (Davis  et  al.,  1989:994). 

Interpreted  within  this  context  of  regret,  these  results  imply  that  during  initial 
exposure  to  the  system,  users  were  concerned  with  overcoming  factors — ^as  defined 
within  that  research  context — ^related  to  ease  of  use  such  that  their  actions  would  no 
longer  lead  to  undesirable  outcomes.  Once  the  users  gained  enough  experience  with  the 
system,  those  factors  which  impacted  their  inability  to  accomplish  their  goals  while  using 
the  system  were  of  greater  concern. 

What  implications  do  the  experiences  of  regret  have  for  our  discussion  of  quality 
of  experience?  Based  on  the  increased  cognitive  availability  of  the  circumstances 
surrounding  a  person’s  inability  to  accomplish  his  or  her  goals,  as  well  as  the  increased 
cognitive  discomfort  caused  by  that  inability,  it  is  reasonable  to  assume  that  many  of  the 
critical  items  pertaining  to  quality  of  experience  may  typify  the  events,  traits,  or  factors 
which  have  influenced  the  relative  success  with  which  the  sampled  Internet  users  were 
able  to  accomplish  their  goals.  Looking  again  at  the  items  themselves,  it  is  not  difficult 
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to  make  such  a  conceptual  leap — ^it  could  even  be  argued  that  all  of  the  critical  quality  of 
experience  items  have  the  potential  to  keep  users  from  attaining  their  goals. 

The  notion  of  regret  may  also  explain  why  the  interface  “enhancing”  features 
afforded  by  many  Web-based  designs  (graphics,  animation,  multimedia  capabilities, 
sound,  3-D  displays,  and  other  frills)  were  not  deemed  important  to  perceptions  of  quality 
of  experience — ^although  the  features  themselves  may  be  aesthetically  pleasing  (and  may 
even  make  the  interface  easier  to  use  in  some  cases),  they  do  little  to  remove  actual 
barriers  to  the  users’  goal  attainment.  Thus,  the  discussion  of  quality  of  experience 
suggests  that  perhaps  some  of  the  effort  and  attention  IT  developers  spend  on  flashy 
graphics,  slick  interfaces,  and  a  myriad  of  multimedia  capabilities  is  premature  if  proper 
consideration  has  not  yet  been  given  to  how  well  the  system  helps  its  users  achieve  their 
goals.  These  conclusions  are  also  consistent  with  the  results  of  a  number  of  TAM-related 
investigations  in  which  usefulness  was  found  to  dominate  ease  of  use  (Adams  et  al., 

1992;  Chau,  1996;  Davis,  1989;  Davis  et  al.,  1989;  Mathieson,  1991;  Szajna,  1996). 

Defining  quality  of  experience  in  terms  of  goal  attainment  also  makes  the 
construct  itself  consistent  with  self-efficacy  theory.  Recall  from  the  previous  discussion 
of  self-efficacy  that  the  theory  suggests  “people  tend  to  avoid  tasks  and  situations  they 
believe  exceed  their  capabilities,  but  they  undertake  and  perform  assuredly  activities  they 
judge  themselves  capable  of  handling”  (Bandura,  1986:393).  If  the  items  which  relate  to 
quality  of  experience  are  such  that  they  have  the  potential  to  keep  people  from 
accomplishing  their  goals,  it  is  reasonable  to  assume  these  items  also  influence  efficacy 
beliefs.  Thus,  items  which  contribute  to  perceptions  of  low  quality  of  experience  may  be 
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as  such  because  they  reduce  efficacy  beliefs  concerning  the  user’s  ability  to  successfully 
use  the  Internet  to  accomplish  his  or  her  goals.  Again,  these  conclusions  are  only 
speculative — ^the  quality  of  experience  construct  must  be  subject  to  further  empirical 
analyses  before  proceeding  from  the  assumption  that  such  cognitive  processes  actually 
have  some  bearing  on  the  determination  of  IT-related  behaviors. 

Looking  Beyond  the  Internet 

While  it  might  seem  difficult  to  extend  the  current  discussion  beyond  Internet- 
type  applications,  the  10  critical  quality  of  experience  items  listed  above  may  not  be 
exclusive  of  other  forms  of  IT.  For  example,  “search  engine  options”  may  simply  be  the 
Internet-based  manifestation  of  more  basic  IT-related  needs  such  as  task-specificity, 
adaptability,  or  flexibility;  “up-to-date  links”  could  just  as  easily  reflect  the  almost 
universal  need  for  accurate,  complete,  and  timely  information,  or  the  need  for  interface 
controls  which  prevent  wasted  efforts  in  searching  for  the  wrong  information. 

Granted,  such  conceptual  leaps  are  tenuous  at  this  initial  stage  of  construct 
development;  however,  there  are  no  reasons  to  assume  that  the  determinants  of  quality  of 
experience  as  it  relates  to  the  Internet  are  not  the  same  (or  derivatives  of  the  same)  factors 
which  influence  perceptions  of  experiential  quality  when  dealing  with  other  forms  or 
applications  of  IT.  Therefore,  applying  the  quality  of  experience  survey,  or  at  least  the 
methodology  used  to  capture  the  relevant  aspects  of  quality  of  experience,  to  other  forms 
of  IT  seems  warranted. 


84 


Conclusion 

Until  the  quality  of  experience  scale  is  related  to  a  relevant  measure  of  IT  use,  the 
utility  of  the  construct  cannot,  as  yet,  be  fully  determined.  Moreover,  some  degree  of 
skepticism  concerning  the  fundamental  aspects  of  the  quality  of  experience  scale,  as  have 
been  suggested  and  defined  within  the  limited  scope  of  this  investigation,  should  be 
maintained  by  the  responsible  and  prudent  investigator.  However,  it  is  hoped  (and  even 
assumed),  amidst  the  myriad  of  suppositions,  inferences,  and  limitations  which  have  yet 
to  be  addressed,  that  there  is  some  value  to  be  gained  by  examining  "quality  of 
experience"  and  its  potential  influence  on  technology  acceptance.  If  this  is  the  case  (and 
there  are  at  least  some  indications  this  is  so),  then  a  great  deal  of  progress  has  already 
been  made. 

Based  on  the  universe  of  potential  items  which  could  conceivably  affect 
perceptions  of  experiential  quality,  the  current  study  produced  a  list  of  items  which  may 
be  more  pertinent  than  others  to  those  perceptions.  Furthermore,  those  items  seem  to  be 
consistent  across  user  groups  of  varying  experience  levels,  possibly  giving  the  IT 
planner/manager  a  means  by  which  to  account  for  otherwise  troublesome  temporal 
changes  in  the  determinants  of  IT  usage.  Certainly,  the  prevailing  cognitive  and  IT- 
related  theories  of  behavioral  determination  provide  some  degree  of  face  validity  for  a 
quality-related  construet  such  as  that  defined  in  this  investigation.  It  is  my  sincere  hope 
that  the  results  of  this  study  have  provided  some  spark  of  theoretical  curiosity  which  in 
itself  justifies  follow-on  research  and  more  controlled  experimentation. 


85 


Appendix  A.  Quality  of  Experience  Survey  TReduced^ 

The  following  few  questions  ask  for  information  about  yourself  and  your  experience  using  the  Internet.  This  information  is  for 
demographic  purposes  only — ^no  attempt  will  be  made  to  link  your  responses  directly  back  to  you. 

Please  indicate  your:  AGE: _ (years)  MAJCOM:  _  RANK/GRADE: _ 

Place  a  check  next  to  the  item  which  best  describes  how  long  you’ve  been  using  the  Internet: 

I’ve  never  used  the  Internet  _  0-6  months  _  6  months  - 1  year 

J‘2years —  2-3years _  More  than  3  years 

**  If  you  have  never  used  the  Internet,  please  indicate  so  in  the  blank  above  and  place  this  questionnaire  back  into  the  return 
envelope  for  mailing;  you  do  not  need  to  complete  the  rest  of  this  survey. 

If  you  checked  any  selection  other  than  "I’ve  never  used  the  Internet,”  proceed  below  to  the  first  set  of  questions. 


Try  to  remember  experiences  you’ve  had  while  using  the  Internet  which  you  would  consider  to  be  **high  quality”  (whatever  that 
means  to  you  personally).  With  those  experiences  in  mind,  use  the  scale  below  and  circle  the  number  corresponding  to  how 
important  you  believe  the  following  items  were  in  creating  that  impression  of  high  quality. 


Not  at  all  Very  Somewhat  Somewhat  Very  Extremely 

Important  Unimportant  Unimportant  Important  Important  Important  Important 


Credibility/integrity  of  information  1 

Relevance  of  information  to  the  task  1 

Access  speed/load  time  1 

Search  engine  options  (ex.  multiple  criteria)  1 

Compatibility  of  browser  (ex.  plug-ins,  frames)  1 
Amount  of  information  available  1 

Relevance  in  search  engine  retums/hits  1 

Organization  of  information  at  web  sites  1 

Ease  of  use  of  browser  1 

Ease  of  use  of  web  site  interface  1 


Reliability  of  connection  (crashes,  disconnects)  1 
Clarity  of  directions  for  navigation  at  web  sites  1 


Completeness  of  web  site  1 

Up-to-date  links  1 

Availability  of  information  at  remote  servers  1 

Access  restrictions  (usenet,  chat,  BBS,  etc.)  1 

Security/privacy  1 

Interesting  presentation  of  information  1 

Advertising  presence  I 


2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 


!!  THIS  QUESTIONNAIRE  IS  CONTINUED  ON  THE  OTHER  SIDE  !! 


\ 
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Try  to  remember  experiences  you’ve  had  while  using  the  Internet  which  you  would  consider  to  be  **low  quality”  (whatever  that 
means  to  you  personally).  With  those  experiences  in  mind,  use  the  scale  below  and  circle  the  number  corresponding  to  how 
important  you  believe  the  following  items  were  in  creating  that  impression  of  low  quality. 


Not  at  all  Very  Somewhat  Somewhat  Very  Extremely 

Important  Unimportant  Unimportant  Important  Important  Important  Important 


Relevance  of  information  to  the  task 

Advertising  presence 

Ease  of  use  of  web  site  interface 

Compatibility  of  browser  (ex.  plug-ins,  frames) 

Security/privacy 

Access  restrictions  (Usenet,  chat,  BBS,  etc.) 
Completeness  of  web  site 
Interesting  presentation  of  information 
Relevance  in  search  engine  retums/hits 
Amount  of  information  available 
Credibility/integrity  of  information 
Access  speed/load  time 
Ease  of  use  of  browser 
Search  engine  options  (ex.  multiple  criteria) 
Clarity  of  directions  for  navigation  at  web  sites 
Availability  of  information  at  remote  servers 
Reliability  of  connection  (crashes,  disconnects) 
Organization  of  information  at  web  sites 
Up-to-date  links 


THANK  YOU  VERY  MUCH  FOR  YOUR  TIME  AND  PARTICIPATION  IN  THIS  STUDY 
PLEASE  PLACE  THIS  COMPLETED  SURVEY  IN  THE  RETURN  ENVELOPE  FOR  MAILING 
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jendix  B.  Inter-item  Correlation  Matrix 


HQ1 

LQ1 

HQ2 

LQ2 

HQ3 

LQ3 

HQ4 

LQ4 

HQ5 

LQ5 

HQ6 

LQ6 

HQ7 

LQ7 

HQ8 

LQ8 

HQ9 

LQ9 

HQ10 

LQ10 

HQ11 

LQ11 

HQ12 

LQ12 

HQ13 

LQ13 

HQ14 

LQ14 

HQ15 

LQ15 

HQ16 

LQ16 

HQ17 

LQ17 

HQ18 

LQ18 

HQ19 

LQ19 


HQ1 


1 

0.363** 

0.556** 

0.272** 

0.07 

0.037 

-0.008 

0.001 

0.052 

0.070 

0.036 

0.160 

-0.042 

-0.018 

0.181* 

0.131 

0.150 

0.151 

0.147 

0.141 

0.096 

0.150 

0.126 

0.026 

0.174* 

0.160 

0.151 

0.241** 

0.128 

0.079 

0.092 

0.222* 

0.163* 

0.099 

-0.064 

0.035 

-0.175* 

-0.046 


Correlation  is 
Correlation  is 


LQ2  |HQ3  1LQ3  |HQ4  |LQ4  |HQ5 
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HQ6 

LQ6 

HQ7 

LQ7 

HQ8 

LQ8 

HQ9 

LQ9 

HQ10 

LQ10 

HQ6 

1 

LQ6 

0.413** 

1 

HQ7 

0.281** 

0.158 

1 

LQ7 

0.137 

0.349** 

mm 

1 

HQ8 

0.208* 

0.109 

1 

LQ8 

0.188* 

0.445** 

0.160 

0.550** 

0.309** 

1 

HQ9 

0.146 

0.176* 

0.158 

0.139 

0.574** 

1 

LQ9 

0.243** 

0.482** 

0.472** 

0.263** 

0.659** 

1 

HQ10 

0.157 

0.246** 

0.426** 

0.714** 

0.276** 

1 

LQ10 

0.347** 

0.432** 

0.336** 

0.551** 

0.259** 

0.565** 

0.294** 

0.638** 

0.400** 

1 

HQ11 

0.249** 

0.146 

0.162 

0.306** 

mm 

0.146 

0.404** 

LQ11 

0.248** 

0.535** 

0.192* 

0.431** 

0.126 

0.522** 

0.237** 

0.528** 

0.213* 

HQ12 

0.281** 

0.247** 

0.298** 

0.361** 

0.350** 

0.356** 

0.416** 

0.457** 

0.456** 

LQ12 

0.128 

0.402** 

0.214* 

0.466** 

0.233** 

0.506** 

0.281** 

0.645** 

0.295** 

HQ13 

0.388** 

0.310** 

0.239** 

0.238** 

0.478** 

0.349** 

iJrKVlil 

0.314** 

0.467** 

LQ13 

0.242** 

0.510** 

0.114 

0.343** 

0.382** 

0.479** 

0.435** 

0.229** 

HQ14 

0.338** 

0.147 

0.171* 

0.353** 

0.263** 

0.318** 

LQ14 

0.166** 

0.534** 

0.111 

0.495** 

0.285** 

0.719** 

0.316** 

0.590** 

0.216* 

0.540** 

HQ15 

0.299** 

0.343** 

0.244** 

0.296** 

0.407** 

0.395** 

0.335** 

0.340** 

0.257** 

0.403** 

LQ15 

0.334** 

0.435** 

0.221* 

0.444** 

0.347** 

0.501** 

0.359** 

0.508** 

0.442** 

0.610** 

HQ16 

0.169* 

0.245** 

0.151 

0.218* 

0.350** 

0.212* 

0.344** 

0.224* 

0.363** 

0.357** 

LQ16 

0.107 

0.295** 

0.060 

0.422** 

0.279** 

0.419** 

0.306** 

0.398** 

0.267** 

0.500** 

HQ17 

0.267** 

0.113 

0.058 

0.027 

0.419** 

0.070 

0.374** 

0.157 

0.373** 

0.276** 

LQ17 

0.239** 

0.286** 

0.061 

0.327** 

0.267** 

0.378** 

0.277** 

0.526** 

0.271** 

0.555** 

HQ18 

0.279** 

0.222* 

0.128 

0.173* 

0.395** 

0.194* 

0.379** 

0.165 

0.298** 

0.291** 

LQ18 

0.226** 

0.419** 

0.155 

0.385** 

0.385** 

0.482** 

0.326** 

0.492** 

0.247** 

0.449** 

HQ19 

0.106 

0.090 

0.106 

0.204* 

0.113 

0.052 

0.122 

0.098 

0.189* 

0.257** 

LQ19 

-0.046 

-0.096 

-0.035 

-0.071 

-0.049 

-0.070 

0.198* 

-0.023 

0.013 

-0.045 

Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*  Correlation  is  significant  at  the  0.05  level  {2-tailed). 


HQ11 

LQ11 

[hQ12 

LQ12 

HQ13 

LQ13 

HQ14 

LQ14 

HQ15 

LQ15 

HQ11 

1 

LQ11 

0.393** 

1 

HQ12 

0.475** 

0.370** 

1 

- 

LQ12 

0.150 

0.451** 

0.530** 

1 

HQ13 

0.399** 

0.268** 

0.584** 

0.298** 

1 

LQ13 

0.166 

0.436** 

0.438** 

0.521** 

1 

HQ14 

0.374** 

0.185* 

0.477** 

0.274** 

0.601** 

1 

LQ14 

0.262** 

0.585** 

0.324** 

0.425** 

0.371** 

0.355** 

1 

HQ15 

0.148 

0.324** 

0.436** 

0.446** 

0.508** 

6.428** 

0.428** 

0.449** 

1 

LQ15 

0.224** 

0.486** 

0.406** 

0.487** 

0.461** 

0.312** 

0.535** 

0.605** 

1 

HQ16 

0.290** 

0.300** 

0.371** 

0.259** 

0.415** 

0.318** 

0.302** 

0.527** 

0.529** 

LQ16 

0.214* 

0.366** 

0.237** 

0.266** 

0.263** 

0.366** 

mm 

0.515** 

0.353** 

0.508** 

HQ17 

0.252** 

0.159 

0.325** 

0.446** 

0.295** 

0.414** 

0.181* 

0.416** 

0.405** 

LQ17 

0.153 

0.452** 

0.365** 

0.426** 

0.357** 

0.471** 

0.405** 

0.502** 

0.417** 

0.598** 

HQ18 

0.149 

0.117 

0.339** 

0.215* 

0.495** 

0.371** 

0.380** 

0.268** 

0.447** 

0.462** 

LQ18 

0.122 

0.289** 

0.380** 

0.502** 

0.395** 

0.599** 

0.294** 

0.513** 

0.545** 

HQ19 

-0.052 

-0.018 

0.267** 

0.193* 

0.239** 

0.086 

0,072 

0.258** 

LQ19 

** 

0.056 

-0.048 

0.152 

_ j. 

0.101 

-0.013  1 

0.000 

QHl 

•*  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


HQ16 

LQ16 

HQ17 

LQ17 

HQ18 

LQ18 

HQ19 

LQ19 

HQ16 

LQ16 

HQ17 

LQ17 

HQ18 

LQ18 

HQ19 

LQ19 

1 

0.556** 

0.380** 

0.306** 

0.350** 

0.255** 

0.381** 

-0.022 

1 

0.329** 

0.561** 

0.174* 

0.262** 

0.204* 

-0.087 

1 

0.562** 

0.449** 

0.342** 

0.234** 

0.025 

1 

0.325** 

0.466** 

0.190* 

-0.097 

1 

0.538** 

0.521** 

0.232** 

1 

0.346** 

0.101 

1 

0.225** 

1 

Correlation  is  significant  at  the  0.01  level  (2-tailed). 
Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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